root/README.html

Revision 2, 21.0 kB (checked in by grig, 7 years ago)

Initial import

Line 
1 <?xml version="1.0" encoding="utf-8" ?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
4 <head>
5 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
6 <meta name="generator" content="Docutils 0.3.9: http://docutils.sourceforge.net/" />
7 <title>Cheesecake: How tasty is your code?</title>
8 <link rel="stylesheet" href="default.css" type="text/css" />
9 </head>
10 <body>
11 <div class="document" id="cheesecake-how-tasty-is-your-code">
12 <h1 class="title">Cheesecake: How tasty is your code?</h1>
13 <div class="contents topic" id="table-of-contents">
14 <p class="topic-title first"><a name="table-of-contents"><strong>Table of Contents</strong></a></p>
15 <ul class="simple">
16 <li><a class="reference" href="#summary" id="id1" name="id1">Summary</a></li>
17 <li><a class="reference" href="#why-cheesecake" id="id2" name="id2">Why Cheesecake?</a></li>
18 <li><a class="reference" href="#usage-examples" id="id3" name="id3">Usage examples</a></li>
19 <li><a class="reference" href="#obtaining-the-source-code" id="id4" name="id4">Obtaining the source code</a></li>
20 <li><a class="reference" href="#mailing-list" id="id5" name="id5">Mailing list</a></li>
21 <li><a class="reference" href="#license" id="id6" name="id6">License</a></li>
22 <li><a class="reference" href="#author-contact-info" id="id7" name="id7">Author contact info</a></li>
23 <li><a class="reference" href="#algorithm-for-computing-the-cheesecake-index" id="id8" name="id8">Algorithm for computing the Cheesecake index</a></li>
24 <li><a class="reference" href="#sample-output" id="id9" name="id9">Sample output</a></li>
25 <li><a class="reference" href="#future-plans" id="id10" name="id10">Future plans</a></li>
26 </ul>
27 </div>
28 <div class="section" id="summary">
29 <h1><a class="toc-backref" href="#id1" name="summary">Summary</a></h1>
30 <p>The idea of the Cheesecake project is to rank Python packages based on various
31 empirical &quot;kwalitee&quot; factors, such as:</p>
32 <blockquote>
33 <ul class="simple">
34 <li>whether the package can be downloaded from PyPI given its name</li>
35 <li>whether the package can be downloaded from a full URL</li>
36 <li>whether the package can be unpacked</li>
37 <li>whether the unpack directory is the same as the package name</li>
38 <li>whether the package can be installed into an alternate directory</li>
39 <li>existence of certain files such as README, INSTALL, LICENSE, setup.py etc.</li>
40 <li>existence of certain directories such as doc, test, demo, examples</li>
41 <li>percentage of modules/functions/classes/methods with docstrings</li>
42 <li>percentage of functions/methods that are unit tested (not currently
43 implemented)</li>
44 <li>average pylint score for all non-test and non-demo modules</li>
45 </ul>
46 </blockquote>
47 <p>Currently, the Cheesecake index is computed for invidual packages obtained
48 through a variety of methods (detailed below). One of the goals of the
49 Cheesecake project is to automatically compute the Cheesecake index for
50 all packages uploaded to the PyPI Cheese Shop (possibly at upload time) and
51 to maintain a collection of Web pages with statistics related to the
52 various indexes of the packages.</p>
53 <p>Cheesecake currently computes 3 types of indexes:</p>
54 <blockquote>
55 <ul class="simple">
56 <li>installability index</li>
57 <li>documentation index</li>
58 <li>code kwalitee index</li>
59 </ul>
60 </blockquote>
61 <p>The algorithms for computing each index type are detailed below.</p>
62 </div>
63 <div class="section" id="why-cheesecake">
64 <h1><a class="toc-backref" href="#id2" name="why-cheesecake">Why Cheesecake?</a></h1>
65 <p>The concept of &quot;kwalitee&quot; originated in the Perl community. Here's a relevant
66 quote:</p>
67 <blockquote>
68 <em>It looks like quality, it sounds like quality, but it's not quite quality.</em></blockquote>
69 <p>Kwalitee is an empiric measure of how good a specific body of code is. It
70 defines quality indicators and measures the code along them. It is currently
71 used by the <a class="reference" href="http://cpants.dev.zsi.at/index.html">CPANTS Testing Service</a>
72 to evaluate the 'goodness' of CPAN packages.</p>
73 <p>Since the Python package repository (aka <a class="reference" href="http://www.python.org/pypi">PyPI</a>)
74 is hosted at the Cheese Shop,
75 it stands to reason that the quality indicator of a PyPI package should be
76 called the Cheesecake index!</p>
77 </div>
78 <div class="section" id="usage-examples">
79 <h1><a class="toc-backref" href="#id3" name="usage-examples">Usage examples</a></h1>
80 <p>To compute the Cheesecake index for a given project, run the cheesecake.py
81 module from the command line and indicate either:</p>
82 <blockquote>
83 <ul class="simple">
84 <li>the package short name (e.g. twill) or</li>
85 <li>the package URL (e.g. <a class="reference" href="http://darcs.idyll.org/~t/projects/twill-0.7.4.tar.gz">http://darcs.idyll.org/~t/projects/twill-0.7.4.tar.gz</a>) or</li>
86 <li>the package path on the file system (e.g. /tmp/twill-latest.tar.gz)</li>
87 </ul>
88 </blockquote>
89 <p>In all cases, the cheesecake module will attempt to download the package
90 if necessary, then to unpack it in a sandbox directory (/tmp/cheesecake_sandbox
91 by default). If either of these operations fails, the Cheesecake index for
92 the package will be 0. If the package can be successfully unpacked, the
93 cheesecake module will compute the values for a variety of indexes detailed
94 in the algorithm given at the end of this file.</p>
95 <p>If the package can be successfully downloaded and unpacked, a log file is
96 created in the sandbox directory and named &lt;package&gt;.log (e.g. the log file
97 for twill-0.7.4.tar.gz is /tmp/cheesecake_sandbox/twill-0.7.4.tar.gz.log).
98 The log file is not automatically deleted after the Cheesecake index is
99 computed, since its purpose is to be inspected for debug information.</p>
100 <p>Command-line examples:</p>
101 <blockquote>
102 <ol class="arabic">
103 <li><p class="first">Compute the Cheesecake index for the Durus package by using setuptools
104 utilities to download the package from PyPI:</p>
105 <pre class="literal-block">
106 python cheesecake.py --name=Durus
107 </pre>
108 </li>
109 <li><p class="first">Compute the Cheesecake index for the Durus package by indicating its URL:</p>
110 <pre class="literal-block">
111 python cheesecake.py --url=http://www.mems-exchange.org/software/durus/Durus-3.1.tar.gz
112 </pre>
113 </li>
114 <li><p class="first">Compute the Cheesecake index for the twill package by indicating its path
115 on the local file system:</p>
116 <pre class="literal-block">
117 python cheesecake.py --path=/tmp/twill-latest.tar.gz
118 </pre>
119 </li>
120 <li><p class="first">To increase the verbosity of the output, use the -v or --verbose option.
121 For more options, run cheesecake.py with -h or --help.</p>
122 </li>
123 </ol>
124 </blockquote>
125 </div>
126 <div class="section" id="obtaining-the-source-code">
127 <h1><a class="toc-backref" href="#id4" name="obtaining-the-source-code">Obtaining the source code</a></h1>
128 <p>The Cheesecake project has not yet been released as a tarball or
129 a Python egg. You can obtain the source code from SourceForge via CVS:</p>
130 <pre class="literal-block">
131 cvs -z3 -d:pserver:anonymous&#64;cvs.sourceforge.net:/cvsroot/cheesecake co -P cheesecake
132 </pre>
133 </div>
134 <div class="section" id="mailing-list">
135 <h1><a class="toc-backref" href="#id5" name="mailing-list">Mailing list</a></h1>
136 <p>Developer mailing list: <a class="reference" href="http://lists.sourceforge.net/lists/listinfo/cheesecake-devel">http://lists.sourceforge.net/lists/listinfo/cheesecake-devel</a></p>
137 </div>
138 <div class="section" id="license">
139 <h1><a class="toc-backref" href="#id6" name="license">License</a></h1>
140 <p>Cheesecake is licensed under the Python Software Foundation license,
141 the same license that governs Python itself. The text of the license is
142 available in the <tt class="docutils literal"><span class="pre">LICENSE</span></tt> file in the source code distribution and
143 can also be downloaded from
144 <a class="reference" href="http://www.opensource.org/licenses/PythonSoftFoundation.php">http://www.opensource.org/licenses/PythonSoftFoundation.php</a>.</p>
145 </div>
146 <div class="section" id="author-contact-info">
147 <h1><a class="toc-backref" href="#id7" name="author-contact-info">Author contact info</a></h1>
148 <p>Grig Gheorghiu</p>
149 <p>Email: &lt;grig at gheorghiu dot net&gt;</p>
150 <p>Web site: <a class="reference" href="http://agiletesting.blogspot.com">http://agiletesting.blogspot.com</a></p>
151 </div>
152 <div class="section" id="algorithm-for-computing-the-cheesecake-index">
153 <h1><a class="toc-backref" href="#id8" name="algorithm-for-computing-the-cheesecake-index">Algorithm for computing the Cheesecake index</a></h1>
154 <p>The cheesecake.py module uses the following constants:</p>
155 <pre class="literal-block">
156 INDEX_PYPI_DOWNLOAD = 50
157 INDEX_PYPI_DISTANCE = 5
158 INDEX_URL_DOWNLOAD  = 25
159 INDEX_UNPACK        = 25
160 INDEX_UNPACK_DIR    = 15
161 INDEX_INSTALL       = 50
162 INDEX_FILE_CRITICAL = 15
163 INDEX_FILE          = 10
164 INDEX_FILE_PYC      = 20
165 INDEX_DIR_CRITICAL  = 25
166 INDEX_DIR           = 20
167 INDEX_DIR_EMPTY     = 5
168
169 MAX_INDEX_DOCSTRINGS = 100 # max. percentage of modules/classes/methods/functions with docstrings
170 MAX_INDEX_PYLINT     = 100 # max. pylint score
171 </pre>
172 <p><strong>Step 0</strong></p>
173 <p>Initialize the Cheesecake index to 0. Also initialize to 0
174 the partial Cheesecake indexes for installability, documentation
175 and code kwalitee.</p>
176 <p>Compute the maximum overall Cheesecake index that can be reached by
177 any given package, which is the sum:</p>
178 <pre class="literal-block">
179 INDEX_PYPI_DOWNLOAD +
180 INDEX_UNPACK + INDEX_UNPACK_DIR +
181 INDEX_INSTALL +
182 MAX_INDEX_DOCSTRINGS + MAX_INDEX_PYLINT +
183 (INDEX_FILE * number_of_expected_files) +
184 (INDEX_FILE_CRITICAL * number_of_expected_critical_files) +
185 (INDEX_DIR * number_of_expected_dirs) +
186 (INDEX_DIR_CRITICAL * number_of_expected_critical_dirs)
187 </pre>
188 <p>Compute the maximum Cheesecake index for installability, which is the sum:</p>
189 <pre class="literal-block">
190 INDEX_PYPI_DOWNLOAD +
191 INDEX_UNPACK + INDEX_UNPACK_DIR +
192 INDEX_INSTALL
193 </pre>
194 <p>Compute the maximum Cheesecake index for documentation, which is the sum:</p>
195 <pre class="literal-block">
196 (INDEX_FILE * number_of_expected_files) +
197 (INDEX_FILE_CRITICAL * number_of_expected_critical_files) +
198 (INDEX_DIR * number_of_expected_dirs) +
199 (INDEX_DIR_CRITICAL * number_of_expected_critical_dirs) +
200 MAX_INDEX_DOCSTRINGS
201 </pre>
202 <p>Compute the maximum Cheesecake index for code kwalitee, which is currently:</p>
203 <pre class="literal-block">
204 MAX_INDEX_PYLINT
205 </pre>
206 <p><strong>Step 1a</strong></p>
207 <p>If short name of the package was specified with <tt class="docutils literal"><span class="pre">-n</span></tt> or <tt class="docutils literal"><span class="pre">--name</span></tt>,
208 try to download the package from the PyPI index page by following the links to
209 the package home page and the package download URL (this is accomplished
210 using setuptools utilities).</p>
211 <p>If not successful, exit with a Cheesecake index of 0. If successful and
212 package was found at the Cheese Shop, add <tt class="docutils literal"><span class="pre">INDEX_PYPI_DOWNLOAD</span></tt> to
213 the overall Cheesecake index and to the installability Cheesecake index.</p>
214 <p>If successful but package was not found at the Cheese Shop, add
215 <tt class="docutils literal"><span class="pre">INDEX_PYPI_DOWNLOAD</span> <span class="pre">-</span> <span class="pre">(INDEX_PYPI_DISTANCE</span> <span class="pre">*</span> <span class="pre">number_of_links_to_package)</span></tt>
216 to the overall Cheesecake index and to the installability Cheesecake index.</p>
217 <p><strong>Step 1b</strong></p>
218 <p>If full URL of the package was specified with <tt class="docutils literal"><span class="pre">-u</span></tt> or <tt class="docutils literal"><span class="pre">--url</span></tt>,
219 try to download the package from the specified URL.</p>
220 <p>If not successful, exit with a Cheesecake index of 0. If successful,
221 add <tt class="docutils literal"><span class="pre">INDEX_URL_DOWNLOAD</span></tt> to the overall Cheesecake index and to
222 the installability Cheesecake index.</p>
223 <p><strong>Step 1c</strong></p>
224 <p>If path to package on local file system was specified with <tt class="docutils literal"><span class="pre">-p</span></tt> or
225 <tt class="docutils literal"><span class="pre">--path</span></tt>, copy the package to the sandbox directory.</p>
226 <p><strong>Step 2</strong></p>
227 <p>Unpack the package (currently supported archive types are zip and
228 tar.gz/tgz; in the near future we will support Python Eggs.)</p>
229 <p>If not successful, exit with a Cheesecake index of 0. If successful, add
230 <tt class="docutils literal"><span class="pre">INDEX_UNPACK</span></tt> to the overall Cheesecake index and to the installability
231 Cheesecake index.</p>
232 <p><strong>Step 3</strong></p>
233 <p>Check that the unpack directory has the same name as the package name
234 (i.e. when unpacking twill-0.7.4.tar.gz, we expect the unpack directory
235 to be twill-0.7.4.)</p>
236 <p>If the unpack directory name is the same as the package name, add
237 <tt class="docutils literal"><span class="pre">INDEX_UNPACK_DIR</span></tt>
238 to the overall Cheesecake index and to the installability Cheesecake index.</p>
239 <p><strong>Step 4</strong></p>
240 <p>Install the package to a temporary directory in a non-default location.
241 If successful, add <tt class="docutils literal"><span class="pre">INDEX_INSTALL</span></tt> to the overall Cheesecake index and to the
242 installability Cheesecake index.</p>
243 <p><strong>Step 5</strong></p>
244 <p>Check for existence of specific files.
245 For each file found, add <tt class="docutils literal"><span class="pre">INDEX_FILE</span></tt> to the overall
246 Cheesecake index and to the documentation Cheesecake index.
247 If the file is deemed critical, add <tt class="docutils literal"><span class="pre">INDEX_FILE_CRITICAL</span></tt> instead.</p>
248 <p>The following special files (&quot;cheese_files&quot;) are currently checked:</p>
249 <pre class="literal-block">
250 cheese_files = [&quot;install&quot;, &quot;changelog&quot;,
251                 &quot;news&quot;, &quot;faq&quot;,
252                 &quot;todo&quot;, &quot;thanks&quot;, &quot;announce&quot;,
253                 &quot;ez_setup.py&quot;,
254                ]
255 </pre>
256 <p>The following files are currently deemed critical:</p>
257 <pre class="literal-block">
258 critical_cheese_files = [&quot;readme&quot;, &quot;license&quot;, &quot;setup.py&quot;]
259 </pre>
260 <p>To check if a file FILE is among the cheese files, the following regular
261 expression is used:</p>
262 <pre class="literal-block">
263 re.search(r&quot;^%s(\.txt)*&quot; % cheese_file, file, re.IGNORECASE)
264 </pre>
265 <p><strong>Step 6</strong></p>
266 <p>Check for existence of specific directories.
267 For each directory found, add <tt class="docutils literal"><span class="pre">INDEX_DIR</span></tt> to the overall Cheesecake
268 index and to the documentation Cheesecake index.
269 If the directory is deemed critical, add <tt class="docutils literal"><span class="pre">INDEX_DIR_CRITICAL</span></tt> instead.
270 If the directory is found empty, add <tt class="docutils literal"><span class="pre">INDEX_DIR_EMPTY</span></tt> instead.</p>
271 <p>The following directories (&quot;cheese_dirs&quot;) are currently checked:</p>
272 <pre class="literal-block">
273 cheese_dirs = [&quot;example&quot;, &quot;demo&quot;]
274 </pre>
275 <p>The following directories are currently deemed critical:</p>
276 <pre class="literal-block">
277 critical_cheese_dirs = [&quot;doc&quot;, &quot;test&quot;]
278 </pre>
279 <p>To check if a directory DIR is among the cheese directories,
280 the following regular expression is used:</p>
281 <pre class="literal-block">
282 re.search(r&quot;^%s&quot; % cheese_dir, DIR, re.ignorecase)
283 </pre>
284 <p><strong>Step 7</strong></p>
285 <p>Check for existence of .pyc files. If found, decrease the score
286 by subtracting <tt class="docutils literal"><span class="pre">INDEX_FILE_PYC</span></tt> from the overall Cheesecake index
287 and from the documentation Cheesecake index.</p>
288 <p><strong>Step 8</strong></p>
289 <p>Compute the percentage of modules/classes/methods/functions that have
290 docstrings associated with them. Only Python modules that are not in test,
291 doc, demo and example directories are checked.
292 Round up the percentage and add it to the overall Cheesecake index and to the
293 documentation Cheesecake index.</p>
294 <p><strong>Step 9</strong></p>
295 <p>If pylint is present on the system, run pylint against all Python files
296 that are not in the test, docs or demo directories.
297 Average the non-negative pylint scores, multiply the average by 10 and
298 add it to the overall Cheesecake index and to the code kwalitee
299 Cheesecake index.</p>
300 <p><strong>Step 10</strong></p>
301 <p>For each of the partial Cheesecake index types (installability,
302 documentation and code kwalitee), display the absolute Cheesecake
303 index for that type as the sum of all indexes of that type computed in
304 the previous steps.
305 Also display the relative Cheesecake index for that type as the percentage
306 of <tt class="docutils literal"><span class="pre">(absolute_index</span> <span class="pre">/</span> <span class="pre">maximum_index)</span></tt>.</p>
307 <p>Display the absolute Cheesecake index for the package as the sum of all
308 indexes computed in the previous steps. Also display the relative Cheesecake
309 index for the package as the percentage of <tt class="docutils literal"><span class="pre">(absolute_index</span> <span class="pre">/</span> <span class="pre">maximum_index)</span></tt>.</p>
310 </div>
311 <div class="section" id="sample-output">
312 <h1><a class="toc-backref" href="#id9" name="sample-output">Sample output</a></h1>
313 <pre class="literal-block">
314 $ python cheesecake.py -n Durus
315 [cheesecake:console] Trying to download package durus from PyPI using setuptools utilities
316 [cheesecake:console] Downloaded package Durus-3.1.tar.gz from http://www.mems-exchange.org/software/durus/Durus-3.1.tar.gz
317 [cheesecake:console] Detailed info available in log file /tmp/cheesecake_sandbox/durus.log
318 [cheesecake:console] A given package can currently reach a MAXIMUM number of 555 points
319 [cheesecake:console] Starting computation of Cheesecake index for package 'Durus-3.1.tar.gz'
320
321 [cheesecake:console] Starting computation of INSTALLABILITY index (max. points = 140)
322 index_pypi_download .....................  45 (downloaded package Durus-3.1.tar.gz following 1 link from PyPI)
323 index_unpack ............................  25 (package untar-ed successfully)
324 index_unpack_dir ........................  15 (unpack directory is Durus-3.1 as expected)
325 index_install ...........................  50 (package installed in /tmp/cheesecake_sandbox/tmp_install_Durus-3.1)
326 ---------------------------------------------
327 INSTALLABILITY INDEX (ABSOLUTE) ......... 135
328 INSTALLABILITY INDEX (RELATIVE) .........  96 (135 out of a maximum of 140 points is 96%)
329
330 [cheesecake:console] Starting computation of DOCUMENTATION index (max. points = 415)
331 index_file_announce .....................   0 (file not found)
332 index_file_changelog ....................   0 (file not found)
333 index_file_ez_setup.py ..................   0 (file not found)
334 index_file_faq ..........................  10 (file found)
335 index_file_install ......................  10 (file found)
336 index_file_license ......................  15 (critical file found)
337 index_file_news .........................   0 (file not found)
338 index_file_readme .......................  15 (critical file found)
339 index_file_setup.py .....................  15 (critical file found)
340 index_file_thanks .......................   0 (file not found)
341 index_file_todo .........................   0 (file not found)
342 index_dir_demo ..........................   0 (directory not found)
343 index_dir_doc ...........................  25 (critical directory found)
344 index_dir_example .......................   0 (directory not found)
345 index_dir_test ..........................  25 (critical directory found)
346 index_docstrings ........................  42 (found 104/249=41.77% modules/classes/methods/functions with docstrings)
347 ---------------------------------------------
348 DOCUMENTATION INDEX (ABSOLUTE) .......... 157
349 DOCUMENTATION INDEX (RELATIVE) ..........  37 (157 out of a maximum of 415 points is 37%)
350
351 [cheesecake:console] Starting computation of CODE KWALITEE index (max. points = 100)
352 index_pylint ............................  64 (average score is 6.30 out of 10)
353 ---------------------------------------------
354 CODE KWALITEE INDEX (ABSOLUTE) ..........  64
355 CODE KWALITEE INDEX (RELATIVE) ..........  64 (64 out of a maximum of 100 points is 64%)
356
357 =============================================
358 OVERALL CHEESECAKE INDEX (ABSOLUTE) ..... 356
359 OVERALL CHEESECAKE INDEX (RELATIVE) .....  64 (356 out of a maximum of 555 points is 64%)
360 </pre>
361 </div>
362 <div class="section" id="future-plans">
363 <h1><a class="toc-backref" href="#id10" name="future-plans">Future plans</a></h1>
364 <p>Cheesecake is under very active development. The immediate goal is to add the unit test
365 index measurement, followed by other metrics inspired from the
366 <a class="reference" href="http://cpants.dev.zsi.at/kwalitee.html">kwalitee indicators</a>.
367 Please edit the <a class="reference" href="http://tracos.org/cheesecake/wiki/IndexMeasurementIdeas">IndexMeasurementIdeas</a>
368 Wiki page to add things that you would like to see covered
369 by the Cheesecake metrics.</p>
370 </div>
371 </div>
372 <div class="footer">
373 <hr class="footer" />
374 Generated with rst2html.py from the
375 <a class="reference" href="http://docutils.sourceforge.net/">docutils</a>
376 distribution. Last modified 2005-12-20 by
377 <a class="reference" href="http://agiletesting.blogspot.com">Grig Gheorghiu</a>.
378 </div>
379 </body>
380 </html>
Note: See TracBrowser for help on using the browser.