Changeset 150

Show
Ignore:
Timestamp:
08/25/06 15:30:34 (2 years ago)
Author:
mk
Message:

Merging mk branch into the trunk.

Files:

Legend:

Unmodified
Added
Removed
Modified
Copied
Moved
  • trunk/README

    r5 r150  
    1212 
    1313 * whether the package can be downloaded from PyPI given its name 
    14  * whether the package can be downloaded from a full URL 
    1514 * whether the package can be unpacked 
    16  * whether the unpack directory is the same as the package name 
    1715 * whether the package can be installed into an alternate directory 
    1816 * existence of certain files such as README, INSTALL, LICENSE, setup.py etc. 
    19  * existence of certain directories such as doc, test, demo, examples 
    2017 * percentage of modules/functions/classes/methods with docstrings 
    21  * percentage of functions/methods that are unit tested (not currently  
    22    implemented) 
    23  * average pylint score for all non-test and non-demo modules 
     18 * pylint score 
     19 * ... and many others 
    2420 
    2521Currently, the Cheesecake index is computed for invidual packages obtained  
     
    7470 
    7571If the package can be successfully downloaded and unpacked, a log file is 
    76 created in the sandbox directory and named <package>.log (e.g. the log file  
    77 for twill-0.7.4.tar.gz is /tmp/cheesecake_sandbox/twill-0.7.4.tar.gz.log). 
    78 The log file is not automatically deleted after the Cheesecake index is 
    79 computed, since its purpose is to be inspected for debug information
     72created in the system /tmp directory and named <package>.log (e.g. the log file  
     73for twill-0.7.4.tar.gz is /tmp/twill-0.7.4.tar.gz.log). 
     74The log file is automatically deleted after the Cheesecake index is 
     75computed, except for situations when errors have occured
    8076 
    8177Command-line examples: 
     
    9894    For more options, run cheesecake.py with -h or --help. 
    9995 
     96Requirements 
     97------------ 
     98 
     99* `pylint <http://www.logilab.org/projects/pylint>`_ is required for 
     100  part of the code kwalitee index computation  
     101* `setuptools <http://peak.telecommunity.com/DevCenter/setuptools>`_ is 
     102  required for the installability index computation 
     103 
    100104Obtaining the source code 
    101105------------------------- 
    102106 
    103 The Cheesecake project has not yet been released as a tarball or 
    104 a Python egg. You can obtain the source code from SourceForge via CVS:: 
    105  
    106   cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/cheesecake co -P cheesecake 
    107  
    108 Mailing list 
    109 ------------ 
    110  
    111 Developer mailing list: http://lists.sourceforge.net/lists/listinfo/cheesecake-devel 
     107You can get the source code via svn:: 
     108 
     109  svn co http://svn.pycheesecake.org/trunk cheesecake 
     110 
     111*Note*: make sure you indicate the target directory when you do the svn checkout, 
     112otherwise the cheesecake package files will be checked out directly in your 
     113current directory. 
     114 
     115You may want to modify your subversion client configuration to automatically 
     116expand tags, like $Id$, $Author$ etc. To do so add following two lines to your 
     117``/.subversion/config``:: 
     118 
     119  enable-auto-props = yes 
     120 
     121in [miscellany] section, and:: 
     122 
     123  *.py = svn:eol-style=native;svn:keywords=Author Date Id Revision 
     124 
     125in [auto-props] section. 
     126 
     127Documentation 
     128------------- 
     129 
     130The most recent code documentation should be always available 
     131at http://agilistas.org/cheesecake/mk/docs/. You can also generate 
     132this documentation directly from the Cheesecake sources. Run this command 
     133from the main source directory:: 
     134 
     135  sh support/generate_docs.sh . 
     136 
     137:Note: Generating documentation requires `epydoc <http://epydoc.sourceforge.net/>`_ 
     138       tool installed. 
     139 
     140Unit tests 
     141---------- 
     142 
     143We use `nose <http://somethingaboutorange.com/mrl/projects/nose/>`_ for automatic 
     144testing of our project, so if you want to test Cheesecake on your machine, please 
     145install that first. Running the standard set of Cheesecake unit test is as easy as:: 
     146 
     147  python setup.py test 
     148 
     149This command is equivalent to:: 
     150 
     151  nosetests --verbose --with-doctest --doctest-tests --include unit --exe 
     152 
     153We also have a set of functional tests, which can be run by issuing this command:: 
     154 
     155  nosetests --verbose --include functional 
     156 
     157Functional tests can take a bit longer to complete, as they test cheesecake_index 
     158script as a whole (as opposed to testing modules and classes separately). 
     159 
     160If you happen to find any of our tests failing, please don't hesitate to contact 
     161us, either via 
     162`cheesecake-devel mailing list <http://lists2.idyll.org/listinfo/cheesecake-dev>`_ 
     163or via `Cheesecake Trac <http://pycheesecake.org/>`_. 
     164 
     165Buildbot 
     166-------- 
     167 
     168A buildbot is happily running svn updates and unit tests. Check it out 
     169`here <http://agilistas.org:8888/>`_. 
     170 
     171Mailing lists 
     172------------- 
     173 
     174* Developer mailing list: http://lists2.idyll.org/listinfo/cheesecake-dev 
     175* User mailing list: http://lists2.idyll.org/listinfo/cheesecake-users 
    112176 
    113177License 
     
    120184http://www.opensource.org/licenses/PythonSoftFoundation.php. 
    121185 
    122 Author contact info 
    123 ------------------- 
     186Authors contact info 
     187-------------------- 
    124188 
    125189Grig Gheorghiu 
    126190 
    127 Email: <grig at gheorghiu dot net> 
    128  
    129 Web site:  http://agiletesting.blogspot.com 
     191:Email: <grig at gheorghiu dot net> 
     192:Web site: http://agiletesting.blogspot.com 
     193 
     194Michal Kwiatkowski 
     195 
     196:Email: <ruby at joker.linuxstuff.pl> 
     197:Web site: http://joker.linuxstuff.pl 
     198 
     199Note: clipart for the cheesecake slice logo used with permission from 
     200Kazumi Hatasa, Director, the Japanese School at Middlebury College, 
     201Purdue University. 
    130202 
    131203Algorithm for computing the Cheesecake index 
    132204-------------------------------------------- 
    133205 
    134 The cheesecake.py module uses the following constants:: 
    135  
    136  INDEX_PYPI_DOWNLOAD = 50 
    137  INDEX_PYPI_DISTANCE = 5 
    138  INDEX_URL_DOWNLOAD  = 25 
    139  INDEX_UNPACK        = 25 
    140  INDEX_UNPACK_DIR    = 15 
    141  INDEX_INSTALL       = 50 
    142  INDEX_FILE_CRITICAL = 15 
    143  INDEX_FILE          = 10 
    144  INDEX_FILE_PYC      = 20 
    145  INDEX_DIR_CRITICAL  = 25 
    146  INDEX_DIR           = 20 
    147  INDEX_DIR_EMPTY     = 5 
    148  
    149  MAX_INDEX_DOCSTRINGS = 100 # max. percentage of modules/classes/methods/functions with docstrings 
    150  MAX_INDEX_PYLINT     = 100 # max. pylint score 
    151  
    152 **Step 0** 
    153  
    154 Initialize the Cheesecake index to 0. Also initialize to 0  
    155 the partial Cheesecake indexes for installability, documentation  
    156 and code kwalitee. 
    157  
    158 Compute the maximum overall Cheesecake index that can be reached by  
    159 any given package, which is the sum:: 
    160  
    161  INDEX_PYPI_DOWNLOAD +  
    162  INDEX_UNPACK + INDEX_UNPACK_DIR +  
    163  INDEX_INSTALL + 
    164  MAX_INDEX_DOCSTRINGS + MAX_INDEX_PYLINT +  
    165  (INDEX_FILE * number_of_expected_files) + 
    166  (INDEX_FILE_CRITICAL * number_of_expected_critical_files) + 
    167  (INDEX_DIR * number_of_expected_dirs) + 
    168  (INDEX_DIR_CRITICAL * number_of_expected_critical_dirs) 
    169  
    170 Compute the maximum Cheesecake index for installability, which is the sum:: 
    171  
    172  INDEX_PYPI_DOWNLOAD +  
    173  INDEX_UNPACK + INDEX_UNPACK_DIR +  
    174  INDEX_INSTALL 
    175  
    176 Compute the maximum Cheesecake index for documentation, which is the sum:: 
    177  
    178  (INDEX_FILE * number_of_expected_files) + 
    179  (INDEX_FILE_CRITICAL * number_of_expected_critical_files) + 
    180  (INDEX_DIR * number_of_expected_dirs) + 
    181  (INDEX_DIR_CRITICAL * number_of_expected_critical_dirs) + 
    182  MAX_INDEX_DOCSTRINGS 
    183  
    184 Compute the maximum Cheesecake index for code kwalitee, which is currently:: 
    185  
    186  MAX_INDEX_PYLINT 
    187  
    188 **Step 1a** 
    189  
    190 If short name of the package was specified with ``-n`` or ``--name``,  
    191 try to download the package from the PyPI index page by following the links to  
    192 the package home page and the package download URL (this is accomplished  
    193 using setuptools utilities). 
    194  
    195 If not successful, exit with a Cheesecake index of 0. If successful and  
    196 package was found at the Cheese Shop, add ``INDEX_PYPI_DOWNLOAD`` to  
    197 the overall Cheesecake index and to the installability Cheesecake index. 
    198  
    199 If successful but package was not found at the Cheese Shop, add  
    200 ``INDEX_PYPI_DOWNLOAD - (INDEX_PYPI_DISTANCE * number_of_links_to_package)`` 
    201 to the overall Cheesecake index and to the installability Cheesecake index. 
    202  
    203 **Step 1b** 
    204  
    205 If full URL of the package was specified with ``-u`` or ``--url``,  
    206 try to download the package from the specified URL. 
    207  
    208 If not successful, exit with a Cheesecake index of 0. If successful,  
    209 add ``INDEX_URL_DOWNLOAD`` to the overall Cheesecake index and to  
    210 the installability Cheesecake index. 
    211  
    212 **Step 1c** 
    213  
    214 If path to package on local file system was specified with ``-p`` or 
    215 ``--path``, copy the package to the sandbox directory. 
    216  
    217 **Step 2** 
    218  
    219 Unpack the package (currently supported archive types are zip and  
    220 tar.gz/tgz; in the near future we will support Python Eggs.) 
    221  
    222 If not successful, exit with a Cheesecake index of 0. If successful, add  
    223 ``INDEX_UNPACK`` to the overall Cheesecake index and to the installability  
    224 Cheesecake index. 
    225  
    226 **Step 3** 
    227  
    228 Check that the unpack directory has the same name as the package name 
    229 (i.e. when unpacking twill-0.7.4.tar.gz, we expect the unpack directory  
    230 to be twill-0.7.4.) 
    231  
    232 If the unpack directory name is the same as the package name, add  
    233 ``INDEX_UNPACK_DIR`` 
    234 to the overall Cheesecake index and to the installability Cheesecake index. 
    235  
    236 **Step 4** 
    237  
    238 Install the package to a temporary directory in a non-default location.  
    239 If successful, add ``INDEX_INSTALL`` to the overall Cheesecake index and to the 
    240 installability Cheesecake index. 
    241  
    242 **Step 5** 
    243  
    244 Check for existence of specific files.  
    245 For each file found, add ``INDEX_FILE`` to the overall  
    246 Cheesecake index and to the documentation Cheesecake index.  
    247 If the file is deemed critical, add ``INDEX_FILE_CRITICAL`` instead. 
    248  
    249 The following special files ("cheese_files") are currently checked:: 
    250  
    251     cheese_files = ["install", "changelog", 
    252                     "news", "faq", 
    253                     "todo", "thanks", "announce", 
    254                     "ez_setup.py", 
    255                    ] 
    256  
    257 The following files are currently deemed critical:: 
    258  
    259     critical_cheese_files = ["readme", "license", "setup.py"] 
    260  
    261 To check if a file FILE is among the cheese files, the following regular 
    262 expression is used:: 
    263  
    264     re.search(r"^%s(\.txt)*" % cheese_file, file, re.IGNORECASE) 
    265  
    266 **Step 6** 
    267  
    268 Check for existence of specific directories.  
    269 For each directory found, add ``INDEX_DIR`` to the overall Cheesecake  
    270 index and to the documentation Cheesecake index.  
    271 If the directory is deemed critical, add ``INDEX_DIR_CRITICAL`` instead. 
    272 If the directory is found empty, add ``INDEX_DIR_EMPTY`` instead. 
    273  
    274 The following directories ("cheese_dirs") are currently checked:: 
    275  
    276     cheese_dirs = ["example", "demo"] 
    277  
    278 The following directories are currently deemed critical:: 
    279  
    280     critical_cheese_dirs = ["doc", "test"] 
    281  
    282 To check if a directory DIR is among the cheese directories,  
    283 the following regular expression is used:: 
    284  
    285     re.search(r"^%s" % cheese_dir, DIR, re.ignorecase) 
    286  
    287 **Step 7** 
    288  
    289 Check for existence of .pyc files. If found, decrease the score  
    290 by subtracting ``INDEX_FILE_PYC`` from the overall Cheesecake index  
    291 and from the documentation Cheesecake index. 
    292  
    293 **Step 8** 
    294  
    295 Compute the percentage of modules/classes/methods/functions that have  
    296 docstrings associated with them. Only Python modules that are not in test,  
    297 doc, demo and example directories are checked.  
    298 Round up the percentage and add it to the overall Cheesecake index and to the 
    299 documentation Cheesecake index. 
    300  
    301 **Step 9** 
    302  
    303 If pylint is present on the system, run pylint against all Python files  
    304 that are not in the test, docs or demo directories.  
    305 Average the non-negative pylint scores, multiply the average by 10 and  
    306 add it to the overall Cheesecake index and to the code kwalitee  
    307 Cheesecake index. 
    308   
    309 **Step 10** 
    310  
    311 For each of the partial Cheesecake index types (installability,  
    312 documentation and code kwalitee), display the absolute Cheesecake  
    313 index for that type as the sum of all indexes of that type computed in  
    314 the previous steps.  
    315 Also display the relative Cheesecake index for that type as the percentage 
    316 of ``(absolute_index / maximum_index)``. 
    317  
    318 Display the absolute Cheesecake index for the package as the sum of all 
    319 indexes computed in the previous steps. Also display the relative Cheesecake 
    320 index for the package as the percentage of ``(absolute_index / maximum_index)``. 
     206The overall Cheesecake score is the sum of values of 3 main indexes 
     207(installability, documentation and code kwalitee). The values of these 
     208indexes rely on values of their subindexes and so on. The whole index tree 
     209and corresponding values for each leaf are presented below: 
     210 
     211* Installability 
     212 
     213  * package is listed on and can be downloaded from PyPI: 50 
     214  * package can be downloaded from given URL: 25 
     215  * package can be unpacked without problems: 25 
     216  * unpacked package directory is the same as package name: 15 
     217  * package has setup.py: 25 
     218  * package can be installed to given directory via "setup.py install": 50 
     219  * package contain generated files, like .pyc: -20 
     220 
     221* Documentation 
     222 
     223  * package contain files listed below 
     224 
     225    * README: 30 
     226    * LICENCE/COPYING: 30 [#oneof]_ 
     227    * ANNOUNCE/CHANGELOG: 20 [#oneof]_ 
     228    * INSTALL: 20 
     229    * AUTHORS: 10 
     230    * FAQ: 10 
     231    * NEWS: 10 
     232    * THANKS: 10 
     233    * TODO: 10 
     234 
     235  * package contain directories listed below 
     236 
     237    * doc/docs: 30 [#oneof]_ 
     238    * test/tests: 30 [#oneof]_ 
     239    * demo/example/examples: 10 [#oneof]_ 
     240 
     241  * code is documented by docstrings: 100 [#docstrings]_ 
     242  * docstrings have proper formatting (like epytext or reST): 30 [#formatted]_ 
     243 
     244* Code Kwalitee 
     245 
     246  * package has high pylint score: 50 
     247  * package has unit tests: 30 
     248 
     249The final score depends on how well the package scores for all indexes 
     250listed above. The score is presented in absolute range (number of points) 
     251and relative (percent of points obtained compared to maximum possible points). 
     252 
     253.. [#oneof] It is enough for a package to contain only one of listed files. 
     254.. [#docstrings] Number of points is proportional to percent of documentable objects 
     255   (module, class or function) that have docstrings. For example, if 
     256   you have 50 documentable objects and 32 of them have docstrings 
     257   your code will get 64 points (because 64% of objects are documented). 
     258.. [#formatted] Number of points depends on number of docstrings that are found 
     259   to contain one of known markup. Currently ReST, epytext and javadoc are 
     260   recognized. We give 10 points for 25% of formatted docstrings, 20 points 
     261   for 50% and 30 points for 75%. 
    321262 
    322263Sample output 
     
    325266:: 
    326267 
    327  $ python cheesecake.py -n Durus 
    328  [cheesecake:console] Trying to download package durus from PyPI using setuptools utilities 
    329  [cheesecake:console] Downloaded package Durus-3.1.tar.gz from http://www.mems-exchange.org/software/durus/Durus-3.1.tar.gz 
    330  [cheesecake:console] Detailed info available in log file /tmp/cheesecake_sandbox/durus.log 
    331  [cheesecake:console] A given package can currently reach a MAXIMUM number of 555 points 
    332  [cheesecake:console] Starting computation of Cheesecake index for package 'Durus-3.1.tar.gz' 
    333  
    334  [cheesecake:console] Starting computation of INSTALLABILITY index (max. points = 140) 
    335  index_pypi_download .....................  45 (downloaded package Durus-3.1.tar.gz following 1 link from PyPI) 
    336  index_unpack ............................  25 (package untar-ed successfully) 
    337  index_unpack_dir ........................  15 (unpack directory is Durus-3.1 as expected) 
    338  index_install ...........................  50 (package installed in /tmp/cheesecake_sandbox/tmp_install_Durus-3.1) 
    339  --------------------------------------------- 
    340  INSTALLABILITY INDEX (ABSOLUTE) ......... 135 
    341  INSTALLABILITY INDEX (RELATIVE) .........  96 (135 out of a maximum of 140 points is 96%) 
    342  
    343  [cheesecake:console] Starting computation of DOCUMENTATION index (max. points = 415) 
    344  index_file_announce .....................   0 (file not found) 
    345  index_file_changelog ....................   0 (file not found) 
    346  index_file_ez_setup.py ..................   0 (file not found) 
    347  index_file_faq ..........................  10 (file found) 
    348  index_file_install ......................  10 (file found) 
    349  index_file_license ......................  15 (critical file found) 
    350  index_file_news .........................   0 (file not found) 
    351  index_file_readme .......................  15 (critical file found) 
    352  index_file_setup.py .....................  15 (critical file found) 
    353  index_file_thanks .......................   0 (file not found) 
    354  index_file_todo .........................   0 (file not found) 
    355  index_dir_demo ..........................   0 (directory not found) 
    356  index_dir_doc ...........................  25 (critical directory found) 
    357  index_dir_example .......................   0 (directory not found) 
    358  index_dir_test ..........................  25 (critical directory found) 
    359  index_docstrings ........................  42 (found 104/249=41.77% modules/classes/methods/functions with docstrings) 
    360  --------------------------------------------- 
    361  DOCUMENTATION INDEX (ABSOLUTE) .......... 157 
    362  DOCUMENTATION INDEX (RELATIVE) ..........  37 (157 out of a maximum of 415 points is 37%) 
    363   
    364  [cheesecake:console] Starting computation of CODE KWALITEE index (max. points = 100) 
    365  index_pylint ............................  64 (average score is 6.30 out of 10) 
    366  --------------------------------------------- 
    367  CODE KWALITEE INDEX (ABSOLUTE) ..........  64 
    368  CODE KWALITEE INDEX (RELATIVE) ..........  64 (64 out of a maximum of 100 points is 64%) 
    369   
    370  ============================================= 
    371  OVERALL CHEESECAKE INDEX (ABSOLUTE) ..... 356 
    372  OVERALL CHEESECAKE INDEX (RELATIVE) .....  64 (356 out of a maximum of 555 points is 64%) 
    373  
     268    $ cheesecake_index -n Durus 
     269    py_pi_download .........................  50  (downloaded package Durus-3.4.1.tar.gz following 1 link from http://www.mems-exchange.org/software/durus/Durus-3.4.1.tar.gz) 
     270    unpack .................................  25  (package unpacked successfully) 
     271    unpack_dir .............................  15  (unpack directory is Durus-3.4.1 as expected) 
     272    setup.py ...............................  25  (setup.py found) 
     273    install ................................  50  (package installed in /tmp/cheesecakeUrZH1A/tmp_install_Durus-3.4.1) 
     274    generated_files ........................   0  (0 .pyc and 0 .pyo files found) 
     275    --------------------------------------------- 
     276    INSTALLABILITY INDEX (ABSOLUTE) ........ 165 
     277    INSTALLABILITY INDEX (RELATIVE) ........ 100  (165 out of a maximum of 165 points is 100%) 
     278 
     279    required_files ......................... 170  (5 files and 2 required directories found) 
     280    docstrings .............................  33  (found 121/369=32.79% objects with docstrings) 
     281    formatted_docstrings ...................   0  (found 6/369=1.63% objects with formatted docstrings) 
     282    --------------------------------------------- 
     283    DOCUMENTATION INDEX (ABSOLUTE) ......... 203 
     284    DOCUMENTATION INDEX (RELATIVE) .........  58  (203 out of a maximum of 350 points is 58%) 
     285 
     286    pylint .................................  33  (pylint score was 6.59 out of 10) 
     287    unit_tested ............................  30  (have unit tests) 
     288    --------------------------------------------- 
     289    CODE KWALITEE INDEX (ABSOLUTE) .........  63 
     290    CODE KWALITEE INDEX (RELATIVE) .........  79  (63 out of a maximum of 80 points is 79%) 
     291 
     292 
     293    ============================================= 
     294    OVERALL CHEESECAKE INDEX (ABSOLUTE) .... 431 
     295    OVERALL CHEESECAKE INDEX (RELATIVE) ....  72  (431 out of a maximum of 595 points is 72%) 
     296 
     297Case study: Cleaning up PyBlosxom 
     298--------------------------------- 
     299 
     300Many thanks to Will Guaraldi for writing 
     301`this article <http://pycheesecake.org/wiki/CleaningUpPyBlosxom>`_ about his 
     302experiences in using Cheesecake to clean up and improve the structure of his 
     303PyBlosxom package. 
     304     
    374305Future plans 
    375306------------ 
     
    377308index measurement, followed by other metrics inspired from the  
    378309`kwalitee indicators <http://cpants.dev.zsi.at/kwalitee.html>`_.  
    379 Please edit the `IndexMeasurementIdeas <http://tracos.org/cheesecake/wiki/IndexMeasurementIdeas>`_ 
     310Please edit the `IndexMeasurementIdeas <http://pycheesecake.org/wiki/IndexMeasurementIdeas>`_ 
    380311Wiki page to add things that you would like to see covered  
    381312by the Cheesecake metrics. 
    382313 
    383 .. footer:: Generated with rst2html.py from the  
    384    `docutils <http://docutils.sourceforge.net/>`_  
    385    distribution. Last modified 2005-12-20 by  
    386    `Grig Gheorghiu <http://agiletesting.blogspot.com>`_. 
     314.. footer:: Last modified 2006-08-21 by `Michal Kwiatkowski <http://joker.linuxstuff.pl>`_. 
  • trunk/cheesecake/__init__.py

    r2 r150  
     1__version__ = '0.6' 
  • trunk/cheesecake/cheesecake_index.py

    r11 r150  
    11#!/usr/bin/env python 
     2"""Cheesecake: How tasty is your code? 
     3 
     4The idea of the Cheesecake project is to rank Python packages based on various  
     5empirical "kwalitee" factors, such as: 
     6 
     7 * whether the package can be downloaded from PyPI given its name 
     8 * whether the package can be unpacked 
     9 * whether the package can be installed into an alternate directory 
     10 * existence of certain files such as README, INSTALL, LICENSE, setup.py etc. 
     11 * percentage of modules/functions/classes/methods with docstrings 
     12 * ... and many others 
    213""" 
    3 Cheesecake: How tasty is your code? 
    4  
    5 The idea of the Cheesecake project is to rank Python packages 
    6 based on various empiric "kwalitee" factors, such as: 
    7  
    8         * whether the package can be downloaded 
    9         * whether the package can be unpacked 
    10         * whether the package can be installed into an alternate directory 
    11         * existence of certain files such as README, INSTALL, LICENSE, setup.py etc. 
    12         * existence of certain directories such as doc, test, demo, examples 
    13         * percentage of modules/functions/classes/methods with docstrings 
    14         * percentage of functions/methods that are unit tested 
    15         * average pylint score for all non-test and non-demo modules 
    16         * whether the package can be unpacked 
    17         * whether the package can be installed into an alternate directory 
    18 """ 
    19  
    20 import os, sys, re, shutil 
    21 import tarfile, zipfile 
     14 
     15import os 
     16import re 
     17import shutil 
     18import sys 
     19import tempfile 
     20 
    2221from optparse import OptionParser 
    2322from urllib import urlretrieve 
     
    2524from math import ceil 
    2625 
    27 from _util import run_cmd, pad_with_dots, pad_left_spaces, pad_msg, pad_line 
    28 from _util import StdoutRedirector 
    2926import logger 
    30 from config import get_pkg_config 
     27 
     28from util import pad_with_dots, pad_left_spaces, pad_right_spaces, pad_msg, pad_line 
     29from util import run_cmd, command_successful 
     30from util import unzip_package, untar_package, unegg_package 
     31from util import mkdirs 
     32from util import StdoutRedirector 
     33from util import time_function 
    3134from codeparser import CodeParser 
     35from cheesecake import __version__ as VERSION 
     36 
     37__docformat__ = 'reStructuredText en' 
     38 
     39 
     40################################################################################ 
     41## Helpers. 
     42################################################################################ 
     43 
     44if 'sorted' not in dir(__builtins__): 
     45    def sorted(L): 
     46        new_list = L[:] 
     47        new_list.sort() 
     48        return new_list 
     49 
     50if 'set' not in dir(__builtins__): 
     51    from sets import Set as set 
     52 
     53def isiterable(obj): 
     54    """Check whether object is iterable. 
     55 
     56    >>> isiterable([1,2,3]) 
     57    True 
     58    >>> isiterable("string") 
     59    True 
     60    >>> isiterable(object) 
     61    False 
     62    """ 
     63    return hasattr(obj, '__iter__') or isinstance(obj, basestring) 
     64 
     65def has_extension(filename, ext): 
     66    """Check if filename has given extension. 
     67 
     68    >>> has_extension("foobar.py", ".py") 
     69    True 
     70    >>> has_extension("foo.bar.py", ".py") 
     71    True 
     72    >>> has_extension("foobar.pyc", ".py") 
     73    False 
     74 
     75    This function is case insensitive. 
     76        >>> has_extension("FOOBAR.PY", ".py") 
     77        True 
     78    """ 
     79    return os.path.splitext(filename.lower())[1] == ext.lower() 
     80 
     81def discover_file_type(filename): 
     82    """Discover type of a file according to its name and its parent directory. 
     83 
     84    Currently supported file types: 
     85        * pyc 
     86        * pyo 
     87        * module: .py files of an application 
     88        * demo: .py files for documentation/demonstration purposes 
     89        * test: .py files used for testing 
     90        * special: .py file for special purposes 
     91 
     92    :Note: This function only check file's name, and doesn't touch the 
     93           filesystem. If you have to, check if file exists by yourself. 
     94 
     95    >>> discover_file_type('module.py') 
     96    'module' 
     97    >>> discover_file_type('./setup.py') 
     98    'special' 
     99    >>> discover_file_type('some/directory/junk.pyc') 
     100    'pyc' 
     101    >>> discover_file_type('examples/readme.txt') 
     102    >>> discover_file_type('examples/runthis.py') 
     103    'demo' 
     104    >>> discover_file_type('optimized.pyo') 
     105    'pyo' 
     106 
     107    >>> test_files = ['ut/test_this_and_that.py', 
     108    ...               'another_test.py', 
     109    ...               'TEST_MY_MODULE.PY'] 
     110    >>> for filename in test_files: 
     111    ...     assert discover_file_type(filename) == 'test', filename 
     112 
     113    >>> discover_file_type('this_is_not_a_test_really.py') 
     114    'module' 
     115    """ 
     116    dirs = filename.split(os.path.sep) 
     117    dirs, filename = dirs[:-1], dirs[-1] 
     118 
     119    if filename in ["setup.py", "ez_setup.py", "__pkginfo__.py"]: 
     120        return 'special' 
     121 
     122    if has_extension(filename, ".pyc"): 
     123        return 'pyc' 
     124    if has_extension(filename, ".pyo"): 
     125        return 'pyo' 
     126    if has_extension(filename, ".py"): 
     127        for dir in dirs: 
     128            if dir in ['test', 'tests']: 
     129                return 'test' 
     130            elif dir in ['doc', 'docs', 'demo', 'example', 'examples']: 
     131                return 'demo' 
     132 
     133        # Most test frameworks look for files starting with "test_". 
     134        # py.test also looks at files with trailing "_test". 
     135        if filename.lower().startswith('test_') or \ 
     136               os.path.splitext(filename)[0].lower().endswith('_test'): 
     137            return 'test' 
     138 
     139        return 'module' 
     140 
     141def get_files_of_type(file_list, file_type): 
     142    """Return files from `file_list` that match given `file_type`. 
     143 
     144    >>> file_list = ['test/test_foo.py', 'setup.py', 'README', 'test/test_bar.py'] 
     145    >>> get_files_of_type(file_list, 'test') 
     146    ['test/test_foo.py', 'test/test_bar.py'] 
     147    """ 
     148    return filter(lambda x: discover_file_type(x) == file_type, file_list) 
     149 
     150def get_package_name_from_path(path): 
     151    """Get package name as file portion of path. 
     152 
     153    >>> get_package_name_from_path('/some/random/path/package.tar.gz') 
     154    'package.tar.gz' 
     155    >>> get_package_name_from_path('/path/underscored_name.zip') 
     156    'underscored_name.zip' 
     157    >>> get_package_name_from_path('/path/unknown.extension.txt') 
     158    'unknown.extension.txt' 
     159    """ 
     160    dir, filename = os.path.split(path) 
     161    return filename 
     162 
     163def get_package_name_from_url(url): 
     164    """Use ``urlparse`` to obtain package name from URL. 
     165 
     166    >>> get_package_name_from_url('http://www.example.com/file.tar.bz2') 
     167    'file.tar.bz2' 
     168    >>> get_package_name_from_url('https://www.example.com/some/dir/file.txt') 
     169    'file.txt' 
     170    """ 
     171    (scheme,location,path,param,query,fragment_id) = urlparse(url) 
     172    return get_package_name_from_path(path) 
     173 
     174def get_package_name_and_type(package, known_extensions): 
     175    """Return package name and type. 
     176 
     177    Package type must exists in known_extensions list. Otherwise None is 
     178    returned. 
     179 
     180    >>> extensions = ['tar.gz', 'zip'] 
     181    >>> get_package_name_and_type('underscored_name.zip', extensions) 
     182    ('underscored_name', 'zip') 
     183    >>> get_package_name_and_type('unknown.extension.txt', extensions) 
     184    """ 
     185    for package_type in known_extensions: 
     186        if package.endswith('.'+package_type): 
     187            # Package name is name of package without file extension (ex. twill-7.3). 
     188            return package[:package.rfind('.'+package_type)], package_type 
     189 
     190def get_method_arguments(method): 
     191    """Return tuple of arguments for given method, excluding self. 
     192 
     193    >>> class Class: 
     194    ...     def method(s, arg1, arg2, other_arg): 
     195    ...         pass 
     196    >>> get_method_arguments(Class.method) 
     197    ('arg1', 'arg2', 'other_arg') 
     198    """ 
     199    return method.func_code.co_varnames[1:method.func_code.co_argcount] 
     200 
     201def get_attributes(obj, names): 
     202    """Return attributes dictionary with keys from `names`. 
     203 
     204    Object is queried for each attribute name, if it doesn't have this 
     205    attribute, default value None will be returned. 
     206 
     207    >>> class Class: 
     208    ...     pass 
     209    >>> obj = Class() 
     210    >>> obj.attr = True 
     211    >>> obj.value = 13 
     212    >>> obj.string = "Hello" 
     213 
     214    >>> d = get_attributes(obj, ['attr', 'string', 'other']) 
     215    >>> d == {'attr': True, 'string': "Hello", 'other': None} 
     216    True 
     217    """ 
     218    attrs = {} 
     219 
     220    for name in names: 
     221        attrs[name] = getattr(obj, name, None) 
     222 
     223    return attrs 
     224 
     225def camel2underscore(name): 
     226    """Convert name from CamelCase to underscore_name. 
     227 
     228    >>> camel2underscore('CamelCase') 
     229    'camel_case' 
     230    >>> camel2underscore('already_underscore_name') 
     231    'already_underscore_name' 
     232    >>> camel2underscore('BigHTMLClass') 
     233    'big_html_class' 
     234    >>> camel2underscore('') 
     235    '' 
     236    """ 
     237    if name and name[0].upper: 
     238        name = name[0].lower() + name[1:] 
     239 
     240    def capitalize(match): 
     241        string = match.group(1).lower().capitalize() 
     242        return string[:-1] + string[-1].upper() 
     243 
     244    def underscore(match): 
     245        return '_' + match.group(1).lower() 
     246 
     247    name = re.sub(r'([A-Z]+)', capitalize, name) 
     248    return re.sub(r'([A-Z])', underscore, name) 
     249 
     250def index_class_to_name(clsname): 
     251    """Covert index class name to index name. 
     252 
     253    >>> index_class_to_name("IndexDownload") 
     254    'download' 
     255    >>> index_class_to_name("IndexUnitTests") 
     256    'unit_tests' 
     257    >>> index_class_to_name("IndexPyPIDownload") 
     258    'py_pi_download' 
     259    """ 
     260    return camel2underscore(clsname.replace('Index', '', 1)) 
     261 
     262def is_empty(path): 
     263    """Returns True if file or directory pointed by `path` is empty. 
     264    """ 
     265    if os.path.isfile(path) and os.path.getsize(path) == 0: 
     266        return True 
     267    if os.path.isdir(path) and os.listdir(path) == []: 
     268        return True 
     269 
     270    return False 
     271 
     272def strip_dir_part(path, root): 
     273    """Strip `root` part from `path`. 
     274 
     275    >>> strip_dir_part('/home/ruby/file', '/home') 
     276    'ruby/file' 
     277    >>> strip_dir_part('/home/ruby/file', '/home/') 
     278    'ruby/file' 
     279    >>> strip_dir_part('/home/ruby/', '/home') 
     280    'ruby/' 
     281    >>> strip_dir_part('/home/ruby/', '/home/') 
     282    'ruby/' 
     283    """ 
     284    path = path.replace(root, '', 1) 
     285 
     286    if path.startswith(os.path.sep): 
     287        path = path[1:] 
     288 
     289    return path 
     290 
     291def get_files_dirs_list(root): 
     292    """Return list of all files and directories below `root`. 
     293 
     294    Root directory is excluded from files/directories paths. 
     295    """ 
     296    files = [] 
     297    directories = [] 
     298 
     299    for dirpath, dirnames, filenames in os.walk(root): 
     300        dirpath = strip_dir_part(dirpath, root) 
     301        files.extend(map(lambda x: os.path.join(dirpath, x), filenames)) 
     302        directories.extend(map(lambda x: os.path.join(dirpath, x), dirnames)) 
     303 
     304    return files, directories 
     305 
     306def length(L): 
     307    """Overall length of all strings in list. 
     308 
     309    >>> length(['a', 'bc', 'd', '', 'efg']) 
     310    7 
     311    """ 
     312    return sum(map(lambda x: len(x), L)) 
     313 
     314def generate_arguments(arguments, max_length): 
     315    """Pass list of strings in chunks of size not greater than max_length. 
     316 
     317    >>> for x in generate_arguments(['abc', 'def'], 4): 
     318    ...     print x 
     319    ['abc'] 
     320    ['def'] 
     321 
     322    >>> for x in generate_arguments(['a', 'bc', 'd', 'e', 'f'], 2): 
     323    ...     print x 
     324    ['a'] 
     325    ['bc'] 
     326    ['d', 'e'] 
     327    ['f'] 
     328 
     329    If a single argument is larger than max_length, ValueError is raised. 
     330        >>> L = [] 
     331        >>> for x in generate_arguments(['abc', 'de', 'fghijk', 'l'], 4): 
     332        ...     L.append(x) 
     333        Traceback (most recent call last): 
     334          ... 
     335        ValueError: Argument 'fghijk' larger than 4. 
     336        >>> L 
     337        [['abc'], ['de']] 
     338    """ 
     339    L = [] 
     340    i = 0 
     341 
     342    # We have to look ahead, so C-style loop here. 
     343    while arguments: 
     344        if L == [] and len(arguments[i]) > max_length: 
     345            raise ValueError("Argument '%s' larger than %d." % (arguments[i], max_length)) 
     346 
     347        L.append(arguments[i]) 
     348 
     349        # End of arguments: yield then terminate. 
     350        if i == len(arguments) - 1: 
     351            yield L 
     352            break 
     353 
     354        # Adding next argument would exceed max_length, so yield now. 
     355        if length(L) + len(arguments[i+1]) > max_length: 
     356            yield L 
     357            L = [] 
     358 
     359        i += 1 
     360 
     361################################################################################ 
     362## Main index class. 
     363################################################################################ 
     364 
     365class NameSetter(type): 
     366    def __init__(cls, name, bases, dict): 
     367        if 'name' not in dict: 
     368            setattr(cls, 'name', name) 
     369 
     370        if 'compute_with' in dict: 
     371            orig_compute_with = cls.compute_with 
     372 
     373            def _timed_compute_with(self, cheesecake): 
     374                (ret, self.time_taken) = time_function(lambda: orig_compute_with(self, cheesecake)) 
     375                self.cheesecake.log.debug("Index %s computed in %.2f seconds." % (self.name, self.time_taken)) 
     376                return ret 
     377 
     378            setattr(cls, 'compute_with', _timed_compute_with) 
     379 
     380    def __repr__(cls): 
     381        return '<Index class: %s>' % cls.name 
     382 
     383def make_indices_dict(indices): 
     384    indices_dict = {} 
     385    for index in indices: 
     386        indices_dict[index.name] = index 
     387    return indices_dict 
    32388 
    33389class Index(object): 
    34     """ 
    35     Encapsulates index attributes such as name, value, details 
    36     """ 
    37  
    38     def __init__(self, type, name="", value=0, details=""): 
    39         self.type = "index_" + type 
    40         self.name = self.type 
    41         if name: self.name += "_" + name 
    42         self.value = value 
    43         self.details = details 
    44          
     390    """Class describing one index. 
     391 
     392    Use it as a container index or subclass to create custom indices. 
     393 
     394    During class initialization, special attribute `name` is magically 
     395    set based on class name. See `NameSetter` definitions for details. 
     396    """ 
     397    __metaclass__ = NameSetter 
     398 
     399    subindices = None 
     400 
     401    name = "unnamed" 
     402    value = -1 
     403    details = "" 
     404    info = "" 
     405 
     406    def __init__(self, *indices): 
     407        # When indices are given explicitly they override the default. 
     408        if indices: 
     409            self.subindices = [] 
     410            self._indices_dict = {} 
     411            for index in indices: 
     412                self.add_subindex(index) 
     413        else: 
     414            if self.subindices: 
     415                new_subindices = [] 
     416                for index in self.subindices: 
     417                    # index must be a class subclassing from Index. 
     418                    assert isinstance(index, type) 
     419                    assert issubclass(index, Index) 
     420                    new_subindices.append(index()) 
     421                self.subindices = new_subindices 
     422            else: 
     423                self.subindices = [] 
     424            # Create dictionary for fast reference. 
     425            self._indices_dict = make_indices_dict(self.subindices) 
     426 
     427        self._compute_arguments = get_method_arguments(self.compute) 
     428 
     429    def _iter_indices(self): 
     430        """Iterate over each subindex and yield their values. 
     431        """ 
     432        for index in self.subindices: 
     433            # Pass Cheesecake instance to other indices. 
     434            yield index.compute_with(self.cheesecake) 
     435            # Print index info after computing. 
     436            if not self.cheesecake.quiet: 
     437                index.print_info() 
     438 
     439    def compute_with(self, cheesecake): 
     440        """Take given Cheesecake instance and compute index value. 
     441        """ 
     442        self.cheesecake = cheesecake 
     443        return self.compute(**get_attributes(cheesecake, self._compute_arguments)) 
     444 
     445    def compute(self): 
     446        """Compute index value and return it. 
     447 
     448        By default this method computes sum of all subindices. Override this 
     449        method when subclassing for different behaviour. 
     450 
     451        Parameters to this function are dynamically prepared with use of 
     452        `get_attributes` function. 
     453 
     454        :Warning: Don't use \*args and \*\*kwds arguments for this method. 
     455        """ 
     456        self.value = sum(self._iter_indices()) 
     457        return self.value 
     458 
     459    def decide(self, cheesecake, when): 
     460        """Decide if this index should be computed. 
     461 
     462        If index has children, it will automatically remove all for which 
     463        decide() return false. 
     464        """ 
     465        if self.subindices: 
     466            # Iterate over copy, as we may remove some elements. 
     467            for index in self.subindices[:]: 
     468                if not getattr(index, 'decide_' + when)(cheesecake): 
     469                    self.remove_subindex(index.name) 
     470            return self.subindices 
     471        return True 
     472 
     473    def decide_before_download(self, cheesecake): 
     474        return self.decide(cheesecake, 'before_download') 
     475 
     476    def decide_after_download(self, cheesecake): 
     477        return self.decide(cheesecake, 'after_download') 
     478 
     479    def add_info(self, info_line): 
     480        """Add information about index computation process, which will 
     481        be visible with --verbose flag. 
     482        """ 
     483        self.info += "[%s] %s\n" % (index_class_to_name(self.name), info_line) 
     484 
     485    def _get_max_value(self): 
     486        if self.subindices: 
     487            return sum(map(lambda index: index.max_value, 
     488                           self.subindices)) 
     489        return 0 
     490 
     491    max_value = property(_get_max_value) 
     492 
     493    def _get_requirements(self): 
     494        if self.subindices: