Changeset 150
- Timestamp:
- 08/25/06 15:30:34 (7 years ago)
- Files:
-
- trunk/AUTHORS (copied) (copied from branches/mk/AUTHORS)
- trunk/CHANGES (copied) (copied from branches/mk/CHANGES)
- trunk/INSTALL (copied) (copied from branches/mk/INSTALL)
- trunk/README (modified) (6 diffs)
- trunk/README.html (deleted)
- trunk/THANKS (copied) (copied from branches/mk/THANKS)
- trunk/__init__.py (deleted)
- trunk/cheesecake/__init__.py (modified) (1 diff)
- trunk/cheesecake/_util.py (deleted)
- trunk/cheesecake/ast_pp.py (copied) (copied from branches/mk/cheesecake/ast_pp.py)
- trunk/cheesecake/cheesecake_index.py (modified) (9 diffs)
- trunk/cheesecake/codeparser.py (modified) (7 diffs)
- trunk/cheesecake/config.py (deleted)
- trunk/cheesecake/model.py (modified) (14 diffs)
- trunk/cheesecake/subprocess.py (modified) (3 diffs)
- trunk/cheesecake/util.py (copied) (copied from branches/mk/cheesecake/util.py)
- trunk/docs (deleted)
- trunk/setup.py (modified) (2 diffs)
- trunk/support (copied) (copied from branches/mk/support)
- trunk/tests/_path_cheesecake.py (deleted)
- trunk/tests/data/import_self.py (copied) (copied from branches/mk/tests/data/import_self.py)
- trunk/tests/data/module1.py (modified) (5 diffs)
- trunk/tests/data/module1.tar.gz (copied) (copied from branches/mk/tests/data/module1.tar.gz)
- trunk/tests/data/package2.tar.gz (copied) (copied from branches/mk/tests/data/package2.tar.gz)
- trunk/tests/data/required.tar.gz (copied) (copied from branches/mk/tests/data/required.tar.gz)
- trunk/tests/data/static.tar.gz (copied) (copied from branches/mk/tests/data/static.tar.gz)
- trunk/tests/functional (copied) (copied from branches/mk/tests/functional)
- trunk/tests/test_code_parser.py (deleted)
- trunk/tests/test_config.py (deleted)
- trunk/tests/test_index_docstrings.py (deleted)
- trunk/tests/test_index_install.py (deleted)
- trunk/tests/test_index_installability.py (deleted)
- trunk/tests/test_index_unpack.py (deleted)
- trunk/tests/test_index_unpack_dir.py (deleted)
- trunk/tests/test_index_url_download.py (deleted)
- trunk/tests/test_init_cleanup.py (deleted)
- trunk/tests/unit (copied) (copied from branches/mk/tests/unit)
Legend:
- Unmodified
- Added
- Removed
- Modified
- Copied
- Moved
trunk/README
r5 r150 12 12 13 13 * whether the package can be downloaded from PyPI given its name 14 * whether the package can be downloaded from a full URL15 14 * whether the package can be unpacked 16 * whether the unpack directory is the same as the package name17 15 * whether the package can be installed into an alternate directory 18 16 * existence of certain files such as README, INSTALL, LICENSE, setup.py etc. 19 * existence of certain directories such as doc, test, demo, examples20 17 * percentage of modules/functions/classes/methods with docstrings 21 * percentage of functions/methods that are unit tested (not currently 22 implemented) 23 * average pylint score for all non-test and non-demo modules 18 * pylint score 19 * ... and many others 24 20 25 21 Currently, the Cheesecake index is computed for invidual packages obtained … … 74 70 75 71 If the package can be successfully downloaded and unpacked, a log file is 76 created in the s andboxdirectory and named <package>.log (e.g. the log file77 for twill-0.7.4.tar.gz is /tmp/ cheesecake_sandbox/twill-0.7.4.tar.gz.log).78 The log file is notautomatically deleted after the Cheesecake index is79 computed, since its purpose is to be inspected for debug information.72 created in the system /tmp directory and named <package>.log (e.g. the log file 73 for twill-0.7.4.tar.gz is /tmp/twill-0.7.4.tar.gz.log). 74 The log file is automatically deleted after the Cheesecake index is 75 computed, except for situations when errors have occured. 80 76 81 77 Command-line examples: … … 98 94 For more options, run cheesecake.py with -h or --help. 99 95 96 Requirements 97 ------------ 98 99 * `pylint <http://www.logilab.org/projects/pylint>`_ is required for 100 part of the code kwalitee index computation 101 * `setuptools <http://peak.telecommunity.com/DevCenter/setuptools>`_ is 102 required for the installability index computation 103 100 104 Obtaining the source code 101 105 ------------------------- 102 106 103 The Cheesecake project has not yet been released as a tarball or 104 a Python egg. You can obtain the source code from SourceForge via CVS:: 105 106 cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/cheesecake co -P cheesecake 107 108 Mailing list 109 ------------ 110 111 Developer mailing list: http://lists.sourceforge.net/lists/listinfo/cheesecake-devel 107 You can get the source code via svn:: 108 109 svn co http://svn.pycheesecake.org/trunk cheesecake 110 111 *Note*: make sure you indicate the target directory when you do the svn checkout, 112 otherwise the cheesecake package files will be checked out directly in your 113 current directory. 114 115 You may want to modify your subversion client configuration to automatically 116 expand tags, like $Id$, $Author$ etc. To do so add following two lines to your 117 ``/.subversion/config``:: 118 119 enable-auto-props = yes 120 121 in [miscellany] section, and:: 122 123 *.py = svn:eol-style=native;svn:keywords=Author Date Id Revision 124 125 in [auto-props] section. 126 127 Documentation 128 ------------- 129 130 The most recent code documentation should be always available 131 at http://agilistas.org/cheesecake/mk/docs/. You can also generate 132 this documentation directly from the Cheesecake sources. Run this command 133 from the main source directory:: 134 135 sh support/generate_docs.sh . 136 137 :Note: Generating documentation requires `epydoc <http://epydoc.sourceforge.net/>`_ 138 tool installed. 139 140 Unit tests 141 ---------- 142 143 We use `nose <http://somethingaboutorange.com/mrl/projects/nose/>`_ for automatic 144 testing of our project, so if you want to test Cheesecake on your machine, please 145 install that first. Running the standard set of Cheesecake unit test is as easy as:: 146 147 python setup.py test 148 149 This command is equivalent to:: 150 151 nosetests --verbose --with-doctest --doctest-tests --include unit --exe 152 153 We also have a set of functional tests, which can be run by issuing this command:: 154 155 nosetests --verbose --include functional 156 157 Functional tests can take a bit longer to complete, as they test cheesecake_index 158 script as a whole (as opposed to testing modules and classes separately). 159 160 If you happen to find any of our tests failing, please don't hesitate to contact 161 us, either via 162 `cheesecake-devel mailing list <http://lists2.idyll.org/listinfo/cheesecake-dev>`_ 163 or via `Cheesecake Trac <http://pycheesecake.org/>`_. 164 165 Buildbot 166 -------- 167 168 A buildbot is happily running svn updates and unit tests. Check it out 169 `here <http://agilistas.org:8888/>`_. 170 171 Mailing lists 172 ------------- 173 174 * Developer mailing list: http://lists2.idyll.org/listinfo/cheesecake-dev 175 * User mailing list: http://lists2.idyll.org/listinfo/cheesecake-users 112 176 113 177 License … … 120 184 http://www.opensource.org/licenses/PythonSoftFoundation.php. 121 185 122 Author contact info123 ------------------- 186 Authors contact info 187 -------------------- 124 188 125 189 Grig Gheorghiu 126 190 127 Email: <grig at gheorghiu dot net> 128 129 Web site: http://agiletesting.blogspot.com 191 :Email: <grig at gheorghiu dot net> 192 :Web site: http://agiletesting.blogspot.com 193 194 Michal Kwiatkowski 195 196 :Email: <ruby at joker.linuxstuff.pl> 197 :Web site: http://joker.linuxstuff.pl 198 199 Note: clipart for the cheesecake slice logo used with permission from 200 Kazumi Hatasa, Director, the Japanese School at Middlebury College, 201 Purdue University. 130 202 131 203 Algorithm for computing the Cheesecake index 132 204 -------------------------------------------- 133 205 134 The cheesecake.py module uses the following constants:: 135 136 INDEX_PYPI_DOWNLOAD = 50 137 INDEX_PYPI_DISTANCE = 5 138 INDEX_URL_DOWNLOAD = 25 139 INDEX_UNPACK = 25 140 INDEX_UNPACK_DIR = 15 141 INDEX_INSTALL = 50 142 INDEX_FILE_CRITICAL = 15 143 INDEX_FILE = 10 144 INDEX_FILE_PYC = 20 145 INDEX_DIR_CRITICAL = 25 146 INDEX_DIR = 20 147 INDEX_DIR_EMPTY = 5 148 149 MAX_INDEX_DOCSTRINGS = 100 # max. percentage of modules/classes/methods/functions with docstrings 150 MAX_INDEX_PYLINT = 100 # max. pylint score 151 152 **Step 0** 153 154 Initialize the Cheesecake index to 0. Also initialize to 0 155 the partial Cheesecake indexes for installability, documentation 156 and code kwalitee. 157 158 Compute the maximum overall Cheesecake index that can be reached by 159 any given package, which is the sum:: 160 161 INDEX_PYPI_DOWNLOAD + 162 INDEX_UNPACK + INDEX_UNPACK_DIR + 163 INDEX_INSTALL + 164 MAX_INDEX_DOCSTRINGS + MAX_INDEX_PYLINT + 165 (INDEX_FILE * number_of_expected_files) + 166 (INDEX_FILE_CRITICAL * number_of_expected_critical_files) + 167 (INDEX_DIR * number_of_expected_dirs) + 168 (INDEX_DIR_CRITICAL * number_of_expected_critical_dirs) 169 170 Compute the maximum Cheesecake index for installability, which is the sum:: 171 172 INDEX_PYPI_DOWNLOAD + 173 INDEX_UNPACK + INDEX_UNPACK_DIR + 174 INDEX_INSTALL 175 176 Compute the maximum Cheesecake index for documentation, which is the sum:: 177 178 (INDEX_FILE * number_of_expected_files) + 179 (INDEX_FILE_CRITICAL * number_of_expected_critical_files) + 180 (INDEX_DIR * number_of_expected_dirs) + 181 (INDEX_DIR_CRITICAL * number_of_expected_critical_dirs) + 182 MAX_INDEX_DOCSTRINGS 183 184 Compute the maximum Cheesecake index for code kwalitee, which is currently:: 185 186 MAX_INDEX_PYLINT 187 188 **Step 1a** 189 190 If short name of the package was specified with ``-n`` or ``--name``, 191 try to download the package from the PyPI index page by following the links to 192 the package home page and the package download URL (this is accomplished 193 using setuptools utilities). 194 195 If not successful, exit with a Cheesecake index of 0. If successful and 196 package was found at the Cheese Shop, add ``INDEX_PYPI_DOWNLOAD`` to 197 the overall Cheesecake index and to the installability Cheesecake index. 198 199 If successful but package was not found at the Cheese Shop, add 200 ``INDEX_PYPI_DOWNLOAD - (INDEX_PYPI_DISTANCE * number_of_links_to_package)`` 201 to the overall Cheesecake index and to the installability Cheesecake index. 202 203 **Step 1b** 204 205 If full URL of the package was specified with ``-u`` or ``--url``, 206 try to download the package from the specified URL. 207 208 If not successful, exit with a Cheesecake index of 0. If successful, 209 add ``INDEX_URL_DOWNLOAD`` to the overall Cheesecake index and to 210 the installability Cheesecake index. 211 212 **Step 1c** 213 214 If path to package on local file system was specified with ``-p`` or 215 ``--path``, copy the package to the sandbox directory. 216 217 **Step 2** 218 219 Unpack the package (currently supported archive types are zip and 220 tar.gz/tgz; in the near future we will support Python Eggs.) 221 222 If not successful, exit with a Cheesecake index of 0. If successful, add 223 ``INDEX_UNPACK`` to the overall Cheesecake index and to the installability 224 Cheesecake index. 225 226 **Step 3** 227 228 Check that the unpack directory has the same name as the package name 229 (i.e. when unpacking twill-0.7.4.tar.gz, we expect the unpack directory 230 to be twill-0.7.4.) 231 232 If the unpack directory name is the same as the package name, add 233 ``INDEX_UNPACK_DIR`` 234 to the overall Cheesecake index and to the installability Cheesecake index. 235 236 **Step 4** 237 238 Install the package to a temporary directory in a non-default location. 239 If successful, add ``INDEX_INSTALL`` to the overall Cheesecake index and to the 240 installability Cheesecake index. 241 242 **Step 5** 243 244 Check for existence of specific files. 245 For each file found, add ``INDEX_FILE`` to the overall 246 Cheesecake index and to the documentation Cheesecake index. 247 If the file is deemed critical, add ``INDEX_FILE_CRITICAL`` instead. 248 249 The following special files ("cheese_files") are currently checked:: 250 251 cheese_files = ["install", "changelog", 252 "news", "faq", 253 "todo", "thanks", "announce", 254 "ez_setup.py", 255 ] 256 257 The following files are currently deemed critical:: 258 259 critical_cheese_files = ["readme", "license", "setup.py"] 260 261 To check if a file FILE is among the cheese files, the following regular 262 expression is used:: 263 264 re.search(r"^%s(\.txt)*" % cheese_file, file, re.IGNORECASE) 265 266 **Step 6** 267 268 Check for existence of specific directories. 269 For each directory found, add ``INDEX_DIR`` to the overall Cheesecake 270 index and to the documentation Cheesecake index. 271 If the directory is deemed critical, add ``INDEX_DIR_CRITICAL`` instead. 272 If the directory is found empty, add ``INDEX_DIR_EMPTY`` instead. 273 274 The following directories ("cheese_dirs") are currently checked:: 275 276 cheese_dirs = ["example", "demo"] 277 278 The following directories are currently deemed critical:: 279 280 critical_cheese_dirs = ["doc", "test"] 281 282 To check if a directory DIR is among the cheese directories, 283 the following regular expression is used:: 284 285 re.search(r"^%s" % cheese_dir, DIR, re.ignorecase) 286 287 **Step 7** 288 289 Check for existence of .pyc files. If found, decrease the score 290 by subtracting ``INDEX_FILE_PYC`` from the overall Cheesecake index 291 and from the documentation Cheesecake index. 292 293 **Step 8** 294 295 Compute the percentage of modules/classes/methods/functions that have 296 docstrings associated with them. Only Python modules that are not in test, 297 doc, demo and example directories are checked. 298 Round up the percentage and add it to the overall Cheesecake index and to the 299 documentation Cheesecake index. 300 301 **Step 9** 302 303 If pylint is present on the system, run pylint against all Python files 304 that are not in the test, docs or demo directories. 305 Average the non-negative pylint scores, multiply the average by 10 and 306 add it to the overall Cheesecake index and to the code kwalitee 307 Cheesecake index. 308 309 **Step 10** 310 311 For each of the partial Cheesecake index types (installability, 312 documentation and code kwalitee), display the absolute Cheesecake 313 index for that type as the sum of all indexes of that type computed in 314 the previous steps. 315 Also display the relative Cheesecake index for that type as the percentage 316 of ``(absolute_index / maximum_index)``. 317 318 Display the absolute Cheesecake index for the package as the sum of all 319 indexes computed in the previous steps. Also display the relative Cheesecake 320 index for the package as the percentage of ``(absolute_index / maximum_index)``. 206 The overall Cheesecake score is the sum of values of 3 main indexes 207 (installability, documentation and code kwalitee). The values of these 208 indexes rely on values of their subindexes and so on. The whole index tree 209 and corresponding values for each leaf are presented below: 210 211 * Installability 212 213 * package is listed on and can be downloaded from PyPI: 50 214 * package can be downloaded from given URL: 25 215 * package can be unpacked without problems: 25 216 * unpacked package directory is the same as package name: 15 217 * package has setup.py: 25 218 * package can be installed to given directory via "setup.py install": 50 219 * package contain generated files, like .pyc: -20 220 221 * Documentation 222 223 * package contain files listed below 224 225 * README: 30 226 * LICENCE/COPYING: 30 [#oneof]_ 227 * ANNOUNCE/CHANGELOG: 20 [#oneof]_ 228 * INSTALL: 20 229 * AUTHORS: 10 230 * FAQ: 10 231 * NEWS: 10 232 * THANKS: 10 233 * TODO: 10 234 235 * package contain directories listed below 236 237 * doc/docs: 30 [#oneof]_ 238 * test/tests: 30 [#oneof]_ 239 * demo/example/examples: 10 [#oneof]_ 240 241 * code is documented by docstrings: 100 [#docstrings]_ 242 * docstrings have proper formatting (like epytext or reST): 30 [#formatted]_ 243 244 * Code Kwalitee 245 246 * package has high pylint score: 50 247 * package has unit tests: 30 248 249 The final score depends on how well the package scores for all indexes 250 listed above. The score is presented in absolute range (number of points) 251 and relative (percent of points obtained compared to maximum possible points). 252 253 .. [#oneof] It is enough for a package to contain only one of listed files. 254 .. [#docstrings] Number of points is proportional to percent of documentable objects 255 (module, class or function) that have docstrings. For example, if 256 you have 50 documentable objects and 32 of them have docstrings 257 your code will get 64 points (because 64% of objects are documented). 258 .. [#formatted] Number of points depends on number of docstrings that are found 259 to contain one of known markup. Currently ReST, epytext and javadoc are 260 recognized. We give 10 points for 25% of formatted docstrings, 20 points 261 for 50% and 30 points for 75%. 321 262 322 263 Sample output … … 325 266 :: 326 267 327 $ python cheesecake.py -n Durus 328 [cheesecake:console] Trying to download package durus from PyPI using setuptools utilities 329 [cheesecake:console] Downloaded package Durus-3.1.tar.gz from http://www.mems-exchange.org/software/durus/Durus-3.1.tar.gz 330 [cheesecake:console] Detailed info available in log file /tmp/cheesecake_sandbox/durus.log 331 [cheesecake:console] A given package can currently reach a MAXIMUM number of 555 points 332 [cheesecake:console] Starting computation of Cheesecake index for package 'Durus-3.1.tar.gz' 333 334 [cheesecake:console] Starting computation of INSTALLABILITY index (max. points = 140) 335 index_pypi_download ..................... 45 (downloaded package Durus-3.1.tar.gz following 1 link from PyPI) 336 index_unpack ............................ 25 (package untar-ed successfully) 337 index_unpack_dir ........................ 15 (unpack directory is Durus-3.1 as expected) 338 index_install ........................... 50 (package installed in /tmp/cheesecake_sandbox/tmp_install_Durus-3.1) 339 --------------------------------------------- 340 INSTALLABILITY INDEX (ABSOLUTE) ......... 135 341 INSTALLABILITY INDEX (RELATIVE) ......... 96 (135 out of a maximum of 140 points is 96%) 342 343 [cheesecake:console] Starting computation of DOCUMENTATION index (max. points = 415) 344 index_file_announce ..................... 0 (file not found) 345 index_file_changelog .................... 0 (file not found) 346 index_file_ez_setup.py .................. 0 (file not found) 347 index_file_faq .......................... 10 (file found) 348 index_file_install ...................... 10 (file found) 349 index_file_license ...................... 15 (critical file found) 350 index_file_news ......................... 0 (file not found) 351 index_file_readme ....................... 15 (critical file found) 352 index_file_setup.py ..................... 15 (critical file found) 353 index_file_thanks ....................... 0 (file not found) 354 index_file_todo ......................... 0 (file not found) 355 index_dir_demo .......................... 0 (directory not found) 356 index_dir_doc ........................... 25 (critical directory found) 357 index_dir_example ....................... 0 (directory not found) 358 index_dir_test .......................... 25 (critical directory found) 359 index_docstrings ........................ 42 (found 104/249=41.77% modules/classes/methods/functions with docstrings) 360 --------------------------------------------- 361 DOCUMENTATION INDEX (ABSOLUTE) .......... 157 362 DOCUMENTATION INDEX (RELATIVE) .......... 37 (157 out of a maximum of 415 points is 37%) 363 364 [cheesecake:console] Starting computation of CODE KWALITEE index (max. points = 100) 365 index_pylint ............................ 64 (average score is 6.30 out of 10) 366 --------------------------------------------- 367 CODE KWALITEE INDEX (ABSOLUTE) .......... 64 368 CODE KWALITEE INDEX (RELATIVE) .......... 64 (64 out of a maximum of 100 points is 64%) 369 370 ============================================= 371 OVERALL CHEESECAKE INDEX (ABSOLUTE) ..... 356 372 OVERALL CHEESECAKE INDEX (RELATIVE) ..... 64 (356 out of a maximum of 555 points is 64%) 373 268 $ cheesecake_index -n Durus 269 py_pi_download ......................... 50 (downloaded package Durus-3.4.1.tar.gz following 1 link from http://www.mems-exchange.org/software/durus/Durus-3.4.1.tar.gz) 270 unpack ................................. 25 (package unpacked successfully) 271 unpack_dir ............................. 15 (unpack directory is Durus-3.4.1 as expected) 272 setup.py ............................... 25 (setup.py found) 273 install ................................ 50 (package installed in /tmp/cheesecakeUrZH1A/tmp_install_Durus-3.4.1) 274 generated_files ........................ 0 (0 .pyc and 0 .pyo files found) 275 --------------------------------------------- 276 INSTALLABILITY INDEX (ABSOLUTE) ........ 165 277 INSTALLABILITY INDEX (RELATIVE) ........ 100 (165 out of a maximum of 165 points is 100%) 278 279 required_files ......................... 170 (5 files and 2 required directories found) 280 docstrings ............................. 33 (found 121/369=32.79% objects with docstrings) 281 formatted_docstrings ................... 0 (found 6/369=1.63% objects with formatted docstrings) 282 --------------------------------------------- 283 DOCUMENTATION INDEX (ABSOLUTE) ......... 203 284 DOCUMENTATION INDEX (RELATIVE) ......... 58 (203 out of a maximum of 350 points is 58%) 285 286 pylint ................................. 33 (pylint score was 6.59 out of 10) 287 unit_tested ............................ 30 (have unit tests) 288 --------------------------------------------- 289 CODE KWALITEE INDEX (ABSOLUTE) ......... 63 290 CODE KWALITEE INDEX (RELATIVE) ......... 79 (63 out of a maximum of 80 points is 79%) 291 292 293 ============================================= 294 OVERALL CHEESECAKE INDEX (ABSOLUTE) .... 431 295 OVERALL CHEESECAKE INDEX (RELATIVE) .... 72 (431 out of a maximum of 595 points is 72%) 296 297 Case study: Cleaning up PyBlosxom 298 --------------------------------- 299 300 Many thanks to Will Guaraldi for writing 301 `this article <http://pycheesecake.org/wiki/CleaningUpPyBlosxom>`_ about his 302 experiences in using Cheesecake to clean up and improve the structure of his 303 PyBlosxom package. 304 374 305 Future plans 375 306 ------------ … … 377 308 index measurement, followed by other metrics inspired from the 378 309 `kwalitee indicators <http://cpants.dev.zsi.at/kwalitee.html>`_. 379 Please edit the `IndexMeasurementIdeas <http:// tracos.org/cheesecake/wiki/IndexMeasurementIdeas>`_310 Please edit the `IndexMeasurementIdeas <http://pycheesecake.org/wiki/IndexMeasurementIdeas>`_ 380 311 Wiki page to add things that you would like to see covered 381 312 by the Cheesecake metrics. 382 313 383 .. footer:: Generated with rst2html.py from the 384 `docutils <http://docutils.sourceforge.net/>`_ 385 distribution. Last modified 2005-12-20 by 386 `Grig Gheorghiu <http://agiletesting.blogspot.com>`_. 314 .. footer:: Last modified 2006-08-21 by `Michal Kwiatkowski <http://joker.linuxstuff.pl>`_. trunk/cheesecake/__init__.py
r2 r150 1 __version__ = '0.6' trunk/cheesecake/cheesecake_index.py
r11 r150 1 1 #!/usr/bin/env python 2 """Cheesecake: How tasty is your code? 3 4 The idea of the Cheesecake project is to rank Python packages based on various 5 empirical "kwalitee" factors, such as: 6 7 * whether the package can be downloaded from PyPI given its name 8 * whether the package can be unpacked 9 * whether the package can be installed into an alternate directory 10 * existence of certain files such as README, INSTALL, LICENSE, setup.py etc. 11 * percentage of modules/functions/classes/methods with docstrings 12 * ... and many others 2 13 """ 3 Cheesecake: How tasty is your code? 4 5 The idea of the Cheesecake project is to rank Python packages 6 based on various empiric "kwalitee" factors, such as: 7 8 * whether the package can be downloaded 9 * whether the package can be unpacked 10 * whether the package can be installed into an alternate directory 11 * existence of certain files such as README, INSTALL, LICENSE, setup.py etc. 12 * existence of certain directories such as doc, test, demo, examples 13 * percentage of modules/functions/classes/methods with docstrings 14 * percentage of functions/methods that are unit tested 15 * average pylint score for all non-test and non-demo modules 16 * whether the package can be unpacked 17 * whether the package can be installed into an alternate directory 18 """ 19 20 import os, sys, re, shutil 21 import tarfile, zipfile 14 15 import os 16 import re 17 import shutil 18 import sys 19 import tempfile 20 22 21 from optparse import OptionParser 23 22 from urllib import urlretrieve … … 25 24 from math import ceil 26 25 27 from _util import run_cmd, pad_with_dots, pad_left_spaces, pad_msg, pad_line28 from _util import StdoutRedirector29 26 import logger 30 from config import get_pkg_config 27 28 from util import pad_with_dots, pad_left_spaces, pad_right_spaces, pad_msg, pad_line 29 from util import run_cmd, command_successful 30 from util import unzip_package, untar_package, unegg_package 31 from util import mkdirs 32 from util import StdoutRedirector 33 from util import time_function 31 34 from codeparser import CodeParser 35 from cheesecake import __version__ as VERSION 36 37 __docformat__ = 'reStructuredText en' 38 39 40 ################################################################################ 41 ## Helpers. 42 ################################################################################ 43 44 if 'sorted' not in dir(__builtins__): 45 def sorted(L): 46 new_list = L[:] 47 new_list.sort() 48 return new_list 49 50 if 'set' not in dir(__builtins__): 51 from sets import Set as set 52 53 def isiterable(obj): 54 """Check whether object is iterable. 55 56 >>> isiterable([1,2,3]) 57 True 58 >>> isiterable("string") 59 True 60 >>> isiterable(object) 61 False 62 """ 63 return hasattr(obj, '__iter__') or isinstance(obj, basestring) 64 65 def has_extension(filename, ext): 66 """Check if filename has given extension. 67 68 >>> has_extension("foobar.py", ".py") 69 True 70 >>> has_extension("foo.bar.py", ".py") 71 True 72 >>> has_extension("foobar.pyc", ".py") 73 False 74 75 This function is case insensitive. 76 >>> has_extension("FOOBAR.PY", ".py") 77 True 78 """ 79 return os.path.splitext(filename.lower())[1] == ext.lower() 80 81 def discover_file_type(filename): 82 """Discover type of a file according to its name and its parent directory. 83 84 Currently supported file types: 85 * pyc 86 * pyo 87 * module: .py files of an application 88 * demo: .py files for documentation/demonstration purposes 89 * test: .py files used for testing 90 * special: .py file for special purposes 91 92 :Note: This function only check file's name, and doesn't touch the 93 filesystem. If you have to, check if file exists by yourself. 94 95 >>> discover_file_type('module.py') 96 'module' 97 >>> discover_file_type('./setup.py') 98 'special' 99 >>> discover_file_type('some/directory/junk.pyc') 100 'pyc' 101 >>> discover_file_type('examples/readme.txt') 102 >>> discover_file_type('examples/runthis.py') 103 'demo' 104 >>> discover_file_type('optimized.pyo') 105 'pyo' 106 107 >>> test_files = ['ut/test_this_and_that.py', 108 ... 'another_test.py', 109 ... 'TEST_MY_MODULE.PY'] 110 >>> for filename in test_files: 111 ... assert discover_file_type(filename) == 'test', filename 112 113 >>> discover_file_type('this_is_not_a_test_really.py') 114 'module' 115 """ 116 dirs = filename.split(os.path.sep) 117 dirs, filename = dirs[:-1], dirs[-1] 118 119 if filename in ["setup.py", "ez_setup.py", "__pkginfo__.py"]: 120 return 'special' 121 122 if has_extension(filename, ".pyc"): 123 return 'pyc' 124 if has_extension(filename, ".pyo"): 125 return 'pyo' 126 if has_extension(filename, ".py"): 127 for dir in dirs: 128 if dir in ['test', 'tests']: 129 return 'test' 130 elif dir in ['doc', 'docs', 'demo', 'example', 'examples']: 131 return 'demo' 132 133 # Most test frameworks look for files starting with "test_". 134 # py.test also looks at files with trailing "_test". 135 if filename.lower().startswith('test_') or \ 136 os.path.splitext(filename)[0].lower().endswith('_test'): 137 return 'test' 138 139 return 'module' 140 141 def get_files_of_type(file_list, file_type): 142 """Return files from `file_list` that match given `file_type`. 143 144 >>> file_list = ['test/test_foo.py', 'setup.py', 'README', 'test/test_bar.py'] 145 >>> get_files_of_type(file_list, 'test') 146 ['test/test_foo.py', 'test/test_bar.py'] 147 """ 148 return filter(lambda x: discover_file_type(x) == file_type, file_list) 149 150 def get_package_name_from_path(path): 151 """Get package name as file portion of path. 152 153 >>> get_package_name_from_path('/some/random/path/package.tar.gz') 154 'package.tar.gz' 155 >>> get_package_name_from_path('/path/underscored_name.zip') 156 'underscored_name.zip' 157 >>> get_package_name_from_path('/path/unknown.extension.txt') 158 'unknown.extension.txt' 159 """ 160 dir, filename = os.path.split(path) 161 return filename 162 163 def get_package_name_from_url(url): 164 """Use ``urlparse`` to obtain package name from URL. 165 166 >>> get_package_name_from_url('http://www.example.com/file.tar.bz2') 167 'file.tar.bz2' 168 >>> get_package_name_from_url('https://www.example.com/some/dir/file.txt') 169 'file.txt' 170 """ 171 (scheme,location,path,param,query,fragment_id) = urlparse(url) 172 return get_package_name_from_path(path) 173 174 def get_package_name_and_type(package, known_extensions): 175 """Return package name and type. 176 177 Package type must exists in known_extensions list. Otherwise None is 178 returned. 179 180 >>> extensions = ['tar.gz', 'zip'] 181 >>> get_package_name_and_type('underscored_name.zip', extensions) 182 ('underscored_name', 'zip') 183 >>> get_package_name_and_type('unknown.extension.txt', extensions) 184 """ 185 for package_type in known_extensions: 186 if package.endswith('.'+package_type): 187 # Package name is name of package without file extension (ex. twill-7.3). 188 return package[:package.rfind('.'+package_type)], package_type 189 190 def get_method_arguments(method): 191 """Return tuple of arguments for given method, excluding self. 192 193 >>> class Class: 194 ... def method(s, arg1, arg2, other_arg): 195 ... pass 196 >>> get_method_arguments(Class.method) 197 ('arg1', 'arg2', 'other_arg') 198 """ 199 return method.func_code.co_varnames[1:method.func_code.co_argcount] 200 201 def get_attributes(obj, names): 202 """Return attributes dictionary with keys from `names`. 203 204 Object is queried for each attribute name, if it doesn't have this 205 attribute, default value None will be returned. 206 207 >>> class Class: 208 ... pass 209 >>> obj = Class() 210 >>> obj.attr = True 211 >>> obj.value = 13 212 >>> obj.string = "Hello" 213 214 >>> d = get_attributes(obj, ['attr', 'string', 'other']) 215 >>> d == {'attr': True, 'string': "Hello", 'other': None} 216 True 217 """ 218 attrs = {} 219 220 for name in names: 221 attrs[name] = getattr(obj, name, None) 222 223 return attrs 224 225 def camel2underscore(name): 226 """Convert name from CamelCase to underscore_name. 227 228 >>> camel2underscore('CamelCase') 229 'camel_case' 230 >>> camel2underscore('already_underscore_name') 231 'already_underscore_name' 232 >>> camel2underscore('BigHTMLClass') 233 'big_html_class' 234 >>> camel2underscore('') 235 '' 236 """ 237 if name and name[0].upper: 238 name = name[0].lower() + name[1:] 239 240 def capitalize(match): 241 string = match.group(1).lower().capitalize() 242 return string[:-1] + string[-1].upper() 243 244 def underscore(match): 245 return '_' + match.group(1).lower() 246 247 name = re.sub(r'([A-Z]+)', capitalize, name) 248 return re.sub(r'([A-Z])', underscore, name) 249 250 def index_class_to_name(clsname): 251 """Covert index class name to index name. 252 253 >>> index_class_to_name("IndexDownload") 254 'download' 255 >>> index_class_to_name("IndexUnitTests") 256 'unit_tests' 257 >>> index_class_to_name("IndexPyPIDownload") 258 'py_pi_download' 259 """ 260 return camel2underscore(clsname.replace('Index', '', 1)) 261 262 def is_empty(path): 263 """Returns True if file or directory pointed by `path` is empty. 264 """ 265 if os.path.isfile(path) and os.path.getsize(path) == 0: 266 return True 267 if os.path.isdir(path) and os.listdir(path) == []: 268 return True 269 270 return False 271 272 def strip_dir_part(path, root): 273 """Strip `root` part from `path`. 274 275 >>> strip_dir_part('/home/ruby/file', '/home') 276 'ruby/file' 277 >>> strip_dir_part('/home/ruby/file', '/home/') 278 'ruby/file' 279 >>> strip_dir_part('/home/ruby/', '/home') 280 'ruby/' 281 >>> strip_dir_part('/home/ruby/', '/home/') 282 'ruby/' 283 """ 284 path = path.replace(root, '', 1) 285 286 if path.startswith(os.path.sep): 287 path = path[1:] 288 289 return path 290 291 def get_files_dirs_list(root): 292 """Return list of all files and directories below `root`. 293 294 Root directory is excluded from files/directories paths. 295 """ 296 files = [] 297 directories = [] 298 299 for dirpath, dirnames, filenames in os.walk(root): 300 dirpath = strip_dir_part(dirpath, root) 301 files.extend(map(lambda x: os.path.join(dirpath, x), filenames)) 302 directories.extend(map(lambda x: os.path.join(dirpath, x), dirnames)) 303 304 return files, directories 305 306 def length(L): 307 """Overall length of all strings in list. 308 309 >>> length(['a', 'bc', 'd', '', 'efg']) 310 7 311 """ 312 return sum(map(lambda x: len(x), L)) 313 314 def generate_arguments(arguments, max_length): 315 """Pass list of strings in chunks of size not greater than max_length. 316 317 >>> for x in generate_arguments(['abc', 'def'], 4): 318 ... print x 319 ['abc'] 320 ['def'] 321 322 >>> for x in generate_arguments(['a', 'bc', 'd', 'e', 'f'], 2): 323 ... print x 324 ['a'] 325 ['bc'] 326 ['d', 'e'] 327 ['f'] 328 329 If a single argument is larger than max_length, ValueError is raised. 330 >>> L = [] 331 >>> for x in generate_arguments(['abc', 'de', 'fghijk', 'l'], 4): 332 ... L.append(x) 333 Traceback (most recent call last): 334 ... 335 ValueError: Argument 'fghijk' larger than 4. 336 >>> L 337 [['abc'], ['de']] 338 """ 339 L = [] 340 i = 0 341 342 # We have to look ahead, so C-style loop here. 343 while arguments: 344 if L == [] and len(arguments[i]) > max_length: 345 raise ValueError("Argument '%s' larger than %d." % (arguments[i], max_length)) 346 347 L.append(arguments[i]) 348 349 # End of arguments: yield then terminate. 350 if i == len(arguments) - 1: 351 yield L 352 break 353 354 # Adding next argument would exceed max_length, so yield now. 355 if length(L) + len(arguments[i+1]) > max_length: 356 yield L 357 L = [] 358 359 i += 1 360 361 ################################################################################ 362 ## Main index class. 363 ################################################################################ 364 365 class NameSetter(type): 366 def __init__(cls, name, bases, dict): 367 if 'name' not in dict: 368 setattr(cls, 'name', name) 369 370 if 'compute_with' in dict: 371 orig_compute_with = cls.compute_with 372 373 def _timed_compute_with(self, cheesecake): 374 (ret, self.time_taken) = time_function(lambda: orig_compute_with(self, cheesecake)) 375 self.cheesecake.log.debug("Index %s computed in %.2f seconds." % (self.name, self.time_taken)) 376 return ret 377 378 setattr(cls, 'compute_with', _timed_compute_with) 379 380 def __repr__(cls): 381 return '<Index class: %s>' % cls.name 382 383 def make_indices_dict(indices): 384 indices_dict = {} 385 for index in indices: 386 indices_dict[index.name] = index 387 return indices_dict 32 388 33 389 class Index(object): 34 """ 35 Encapsulates index attributes such as name, value, details 36 """ 37 38 def __init__(self, type, name="", value=0, details=""): 39 self.type = "index_" + type 40 self.name = self.type 41 if name: self.name += "_" + name 42 self.value = value 43 self.details = details 44 390 """Class describing one index. 391 392 Use it as a container index or subclass to create custom indices. 393 394 During class initialization, special attribute `name` is magically 395 set based on class name. See `NameSetter` definitions for details. 396 """ 397 __metaclass__ = NameSetter 398 399 subindices = None 400 401 name = "unnamed" 402 value = -1 403 details = "" 404 info = "" 405 406 def __init__(self, *indices): 407 # When indices are given explicitly they override the default. 408 if indices: 409 self.subindices = [] 410 self._indices_dict = {} 411 for index in indices: 412 self.add_subindex(index) 413 else: 414 if self.subindices: 415 new_subindices = [] 416 for index in self.subindices: 417 # index must be a class subclassing from Index. 418 assert isinstance(index, type) 419 assert issubclass(index, Index) 420 new_subindices.append(index()) 421 self.subindices = new_subindices 422 else: 423 self.subindices = [] 424 # Create dictionary for fast reference. 425 self._indices_dict = make_indices_dict(self.subindices) 426 427 self._compute_arguments = get_method_arguments(self.compute) 428 429 def _iter_indices(self): 430 """Iterate over each subindex and yield their values. 431 """ 432 for index in self.subindices: 433 # Pass Cheesecake instance to other indices. 434 yield index.compute_with(self.cheesecake) 435 # Print index info after computing. 436 if not self.cheesecake.quiet: 437 index.print_info() 438 439 def compute_with(self, cheesecake): 440 """Take given Cheesecake instance and compute index value. 441 """ 442 self.cheesecake = cheesecake 443 return self.compute(**get_attributes(cheesecake, self._compute_arguments)) 444 445 def compute(self): 446 """Compute index value and return it. 447 448 By default this method computes sum of all subindices. Override this 449 method when subclassing for different behaviour. 450 451 Parameters to this function are dynamically prepared with use of 452 `get_attributes` function. 453 454 :Warning: Don't use \*args and \*\*kwds arguments for this method. 455 """ 456 self.value = sum(self._iter_indices()) 457 return self.value 458 459 def decide(self, cheesecake, when): 460 """Decide if this index should be computed. 461 462 If index has children, it will automatically remove all for which 463 decide() return false. 464 """ 465 if self.subindices: 466 # Iterate over copy, as we may remove some elements. 467 for index in self.subindices[:]: 468 if not getattr(index, 'decide_' + when)(cheesecake): 469 self.remove_subindex(index.name) 470 return self.subindices 471 return True 472 473 def decide_before_download(self, cheesecake): 474 return self.decide(cheesecake, 'before_download') 475 476 def decide_after_download(self, cheesecake): 477 return self.decide(cheesecake, 'after_download') 478 479 def add_info(self, info_line): 480 """Add information about index computation process, which will 481 be visible with --verbose flag. 482 """ 483 self.info += "[%s] %s\n" % (index_class_to_name(self.name), info_line) 484 485 def _get_max_value(self): 486 if self.subindices: 487 return sum(map(lambda index: index.max_value, 488 self.subindices)) 489 return 0 490 491 max_value = property(_get_max_value) 492 493 def _get_requirements(self): 494 if self.subindices: 495 return list(self._compute_arguments) + \ 496 reduce(lambda x,y: x + y.requirements, self.subindices, []) 497 return list(self._compute_arguments) 498 499 requirements = property(_get_requirements) 500 501 def add_subindex(self, index): 502 """Add subindex. 503 504 :Parameters: 505 `index` : Index instance 506 Index instance for inclusion. 507 """ 508 if not isinstance(index, Index): 509 raise ValueError("subindex have to be instance of Index") 510 511 self.subindices.append(index) 512 self._indices_dict[index.name] = index 513 514 def remove_subindex(self, index_name): 515 """Remove subindex (refered by name). 516 517 :Parameters: 518 `index` : Index name 519 Index name to be removed. 520 """ 521 index = self._indices_dict[index_name] 522 self.subindices.remove(index) 523 del self._indices_dict[index_name] 524 525 def _print_info_one(self): 526 if self.cheesecake.verbose: 527 sys.stdout.write(self.get_info()) 528 print "%s (%s)" % (pad_msg(index_class_to_name(self.name), self.value), self.details) 529 530 def _print_info_many(self): 531 max_value = self.max_value 532 if max_value == 0: 533 return 534 535 percentage = int(ceil(float(self.value) / float(max_value) * 100)) 536 print pad_line("-") 537 538 print pad_msg("%s INDEX (ABSOLUTE)" % self.name, self.value) 539 msg = pad_msg("%s INDEX (RELATIVE)" % self.name, percentage) 540 msg += " (%d out of a maximum of %d points is %d%%)" %\ 541 (self.value, max_value, percentage) 542 543 print msg 544 print 545 45 546 def print_info(self): 46 """ 47 Print index name padded with dots, followed by value and details 48 """ 49 msg = pad_with_dots(self.name) 50 msg += pad_left_spaces(self.value) 51 msg += " (" + self.details + ")" 52 print msg 53 54 class CompositeIndex(object): 55 """ 56 Collection of indexes of same type (e.g. files, dirs) 57 """ 58 59 def __init__(self, type): 60 """ 61 Indexes is a dict mapping names to Index objects 62 """ 63 self.type = type 64 self.indexes = {} 65 66 def set_index(self, name, value=0, details=""): 67 """ 68 Create new index or update existing index with specified attributes 69 """ 70 if self.indexes.has_key(name): 71 index = self.indexes[name] 72 index.value = value 73 index.details = details 547 """Print index name padded with dots, followed by value and details. 548 """ 549 if self.subindices: 550 self._print_info_many() 74 551 else: 75 self.indexes[name] = Index(self.type, name, value, details) 76 77 def print_info(self): 78 """ 79 Print index info for all indexes sorted alphanumerically by name 80 """ 81 names = self.indexes.keys() 82 names.sort() 83 for name in names: 84 index = self.indexes[name] 85 index.print_info() 86 87 def get_value(self): 88 """ 89 Return sum of individual index values 90 """ 552 self._print_info_one() 553 554 def __getitem__(self, name): 555 return self._indices_dict[name] 556 557 def get_info(self): 558 if self.subindices: 559 return ''.join(map(lambda index: index.get_info(), self.subindices)) 560 return self.info 561 562 ################################################################################ 563 ## Index that computes scores based on files and directories. 564 ################################################################################ 565 566 class OneOf(object): 567 def __init__(self, *possibilities): 568 self.possibilities = possibilities 569 def __str__(self): 570 return '/'.join(map(lambda x: str(x), self.possibilities)) 571 572 def WithOptionalExt(name, extensions): 573 """Handy way of writing Cheese rules for files with extensions. 574 575 Instead of writing: 576 >>> one_of = OneOf('readme', 'readme.html', 'readme.txt') 577 578 Write this: 579 >>> opt_ext = WithOptionalExt('readme', ['html', 'txt']) 580 581 It means the same! (representation have a meaning) 582 >>> str(one_of) == str(opt_ext) 583 True 584 """ 585 possibilities = [name] 586 possibilities.extend(map(lambda x: name + '.' + x, extensions)) 587 588 return OneOf(*possibilities) 589 590 def Doc(name): 591 return WithOptionalExt(name, ['html', 'txt']) 592 593 class FilesIndex(Index): 594 _used_rules = [] 595 596 def _compute_from_rules(self, files_list, package_dir, files_rules): 597 self._used_rules = [] 598 files_count = 0 91 599 value = 0 92 for key in self.indexes.keys(): 93 index = self.indexes[key] 94 value += index.value 95 return value 96 97 value = property(get_value) 600 601 for filename in files_list: 602 if not is_empty(os.path.join(package_dir, filename)): 603 score = self.get_score(os.path.basename(filename), files_rules) 604 if score != 0: 605 value += score 606 files_count += 1 607 608 return files_count, value 609 610 def get_score(self, name, specs): 611 for entry, value in specs.iteritems(): 612 if self.match_filename(name, entry): 613 self.cheesecake.log.debug("%d points entry found: %s (%s)" % \ 614 (value, name, entry)) 615 return value 616 617 return 0 618 619 def get_not_used(self, files_rules): 620 """Get only these of files_rules that didn't match during computation. 621 622 >>> rules = { 623 ... Doc('readme'): 30, 624 ... OneOf(Doc('license'), Doc('copying')): 30, 625 ... 'demo': 10, 626 ... } 627 >>> index = FilesIndex() 628 >>> index._used_rules.append('demo') 629 >>> map(lambda x: str(x), index.get_not_used(rules.keys())) 630 ['license/license.html/license.txt/copying/copying.html/copying.txt', 'readme/readme.html/readme.txt'] 631 """ 632 return filter(lambda rule: rule not in self._used_rules, 633 files_rules) 634 635 def match_filename(self, name, rule): 636 """Check if `name` matches given `rule`. 637 """ 638 def equal(x, y): 639 x_root, x_ext = os.path.splitext(x) 640 y_root, y_ext = os.path.splitext(y.lower()) 641 if x_root in [y_root.lower(), y_root.upper(), y_root.capitalize()] \ 642 and x_ext in [y_ext.lower(), y_ext.upper()]: 643 return True 644 return False 645 646 if rule in self._used_rules: 647 return False 648 649 if isinstance(rule, basestring): 650 if equal(name, rule): 651 self._used_rules.append(rule) 652 return True 653 elif isinstance(rule, OneOf): 654 for poss in rule.possibilities: 655 if self.match_filename(name, poss): 656 self._used_rules.append(rule) 657 return True 658 659 return False 660 661 ################################################################################ 662 ## Installability index. 663 ################################################################################ 664 665 class IndexUrlDownload(Index): 666 """Give points for successful downloading of a package. 667 """ 668 max_value = 25 669 670 def compute(self, downloaded_from_url, package, url): 671 if downloaded_from_url: 672 self.details = "downloaded package %s from URL %s" % (package, url) 673 self.value = self.max_value 674 else: 675 self.value = 0 676 677 return self.value 678 679 def decide_before_download(self, cheesecake): 680 return cheesecake.url 681 682 class IndexUnpack(Index): 683 """Give points for successful unpacking of a package archive. 684 """ 685 max_value = 25 686 687 def compute(self, unpacked): 688 if unpacked: 689 self.details = "package unpacked successfully" 690 self.value = self.max_value 691 else: 692 self.details = "package couldn't be unpacked" 693 self.value = 0 694 695 return self.value 696 697 class IndexUnpackDir(Index): 698 """Check if package unpack directory resembles package archive name. 699 """ 700 max_value = 15 701 702 def compute(self, unpack_dir, original_package_name): 703 self.details = "unpack directory is " + unpack_dir 704 705 if original_package_name: 706 self.details += " instead of the expected " + original_package_name 707 self.value = 0 708 else: 709 self.details += " as expected" 710 self.value = self.max_value 711 712 return self.value 713 714 def decide_after_download(self, cheesecake): 715 return cheesecake.package_type != 'egg' 716 717 class IndexSetupPy(FilesIndex): 718 """Reward packages that have setup.py file. 719 """ 720 name = "setup.py" 721 max_value = 25 722 723 files_rules = { 724 'setup.py': 25, 725 } 726 727 def compute(self, files_list, package_dir): 728 setup_py_found, self.value = self._compute_from_rules(files_list, package_dir, self.files_rules) 729 730 if setup_py_found: 731 self.details = "setup.py found" 732 else: 733 self.details = "setup.py not found" 734 735 return self.value 736 737 def decide_after_download(self, cheesecake): 738 return cheesecake.package_type != 'egg' 739 740 class IndexInstall(Index): 741 """Check if package can be installed via "python setup.py" command. 742 """ 743 max_value = 50 744 745 def compute(self, installed, sandbox_install_dir): 746 if installed: 747 self.details = "package installed in %s" % sandbox_install_dir 748 self.value = self.max_value 749 else: 750 self.details = "could not install package in %s" % sandbox_install_dir 751 self.value = 0 752 753 return self.value 754 755 def decide_before_download(self, cheesecake): 756 return not cheesecake.static_only 757 758 class IndexPyPIDownload(Index): 759 """Check if package was successfully downloaded from PyPI 760 and how far from it actual package was. 761 762 Distance is number of links user have to follow to download 763 a given software package. 764 """ 765 max_value = 50 766 distance_penalty = -5 767 768 def compute(self, package, found_on_cheeseshop, found_locally, distance_from_pypi, download_url): 769 if download_url: 770 self.value = self.max_value 771 772 self.details = "downloaded package " + package 773 774 if not found_on_cheeseshop: 775 self.value += (distance_from_pypi - 1) * self.distance_penalty 776 777 if distance_from_pypi: 778 self.details += " following %d link" % distance_from_pypi 779 if distance_from_pypi > 1: 780 self.details += "s" 781 self.details += " from PyPI" 782 else: 783 self.details += " from " + download_url 784 else: 785 self.details += " directly from the Cheese Shop" 786 else: 787 if found_locally: 788 self.details = "found on local filesystem" 789 self.value = 0 790 791 return self.value 792 793 def decide_before_download(self, cheesecake): 794 return cheesecake.name 795 796 class IndexGeneratedFiles(Index): 797 """Lower score for automatically generated files that should 798 not be present in a package. 799 """ 800 generated_files_penalty = -20 801 max_value = 0 802 803 def compute(self, files_list): 804 self.value = 0 805 806 pyc_files = len(get_files_of_type(files_list, 'pyc')) 807 pyo_files = len(get_files_of_type(files_list, 'pyo')) 808 809 if pyc_files > 0 or pyo_files > 0: 810 self.value += self.generated_files_penalty 811 812 self.details = "%d .pyc and %d .pyo files found" % \ 813 (pyc_files, pyo_files) 814 815 return self.value 816 817 def decide_after_download(self, cheesecake): 818 return cheesecake.package_type != 'egg' 819 820 class IndexInstallability(Index): 821 name = "INSTALLABILITY" 822 823 subindices = [ 824 IndexPyPIDownload, 825 IndexUrlDownload, 826 IndexUnpack, 827 IndexUnpackDir, 828 IndexSetupPy, 829 IndexInstall, 830 IndexGeneratedFiles, 831 ] 832 833 ################################################################################ 834 ## Documentation index. 835 ################################################################################ 836 837 class IndexRequiredFiles(FilesIndex): 838 """Check for existence of important files, like README or INSTALL. 839 """ 840 cheese_files = { 841 Doc('readme'): 30, 842 OneOf(Doc('license'), Doc('copying')): 30, 843 844 OneOf(Doc('announce'), Doc('changelog'), Doc('changes')): 20, 845 Doc('install'): 20, 846 847 Doc('authors'): 10, 848 Doc('faq'): 10, 849 Doc('news'): 10, 850 Doc('thanks'): 10, 851 Doc('todo'): 10, 852 } 853 854 cheese_dirs = { 855 OneOf('doc', 'docs'): 30, 856 OneOf('test', 'tests'): 30, 857 858 OneOf('demo', 'example', 'examples'): 10, 859 } 860 861 max_value = sum(cheese_files.values() + cheese_dirs.values()) 862 863 def compute(self, files_list, dirs_list, package_dir): 864 # Inform user of files and directories the package is missing. 865 def make_info(dictionary, what): 866 missing = self.get_not_used(dictionary.keys()) 867 importance = {30: ' critical', 20: ' important'} 868 869 positive_msg = "Package has%s %s: %s." 870 negative_msg = "Package doesn't have%s %s: %s." 871 872 for key in dictionary.keys(): 873 msg = positive_msg 874 if key in missing: 875 msg = negative_msg 876 self.add_info(msg % (importance.get(dictionary[key], ''), what, str(key))) 877 878 # Compute required files. 879 files_count, files_value = self._compute_from_rules(files_list, package_dir, self.cheese_files) 880 make_info(self.cheese_files, 'file') 881 882 # Compute required directories. 883 dirs_count, dirs_value = self._compute_from_rules(dirs_list, package_dir, self.cheese_dirs) 884 make_info(self.cheese_dirs, 'directory') 885 886 self.value = files_value + dirs_value 887 888 self.details = "%d files and %d required directories found" % \ 889 (files_count, dirs_count) 890 891 return self.value 892 893 class IndexDocstrings(Index): 894 """Compute how many objects have relevant docstrings. 895 """ 896 max_value = 100 897 898 def compute(self, object_cnt, docstring_cnt): 899 percent = 0 900 if object_cnt > 0: 901 percent = float(docstring_cnt)/float(object_cnt) 902 903 # Scale the result. 904 self.value = int(ceil(percent * self.max_value)) 905 906 self.details = "found %d/%d=%.2f%% objects with docstrings" %\ 907 (docstring_cnt, object_cnt, percent*100) 908 909 return self.value 910 911 class IndexFormattedDocstrings(Index): 912 """Compute how many of existing docstrings include any formatting, 913 like epytext or reST. 914 """ 915 max_value = 30 916 917 def compute(self, object_cnt, docformat_cnt): 918 percent = 0 919 if object_cnt > 0: 920 percent = float(docformat_cnt)/float(object_cnt) 921 922 # Scale the result. 923 # We give 10p for 25% of formatted docstrings, 20p for 50% and 30p for 75%. 924 self.value = 0 925 if percent > 0.75: 926 self.add_info("%.2f%% formatted docstrings found, which is > 75%% and is worth 30p." % (percent*100)) 927 self.value = 30 928 elif percent > 0.50: 929 self.add_info("%.2f%% formatted docstrings found, which is > 50%% and is worth 20p." % (percent*100)) 930 self.value = 20 931 elif percent > 0.25: 932 self.add_info("%.2f%% formatted docstrings found, which is > 25%% and is worth 10p." % (percent*100)) 933 self.value = 10 934 else: 935 self.add_info("%.2f%% formatted docstrings found, which is < 25%%, no points given." % (percent*100)) 936 937 self.details = "found %d/%d=%.2f%% objects with formatted docstrings" %\ 938 (docformat_cnt, object_cnt, percent*100) 939 940 return self.value 941 942 class IndexDocumentation(Index): 943 name = "DOCUMENTATION" 944 945 subindices = [ 946 IndexRequiredFiles, 947 IndexDocstrings, 948 IndexFormattedDocstrings, 949 ] 950 951 ################################################################################ 952 ## Code "kwalitee" index. 953 ################################################################################ 954 955 class IndexUnitTests(Index): 956 """Compute unittest index as percentage of methods/functions 957 that are exercised in unit tests. 958 """ 959 max_value = 50 960 961 def compute(self, files_list, functions, classes, package_dir): 962 unittest_cnt = 0 963 functions_tested = set() 964 965 # Gather all function names called from test files. 966 for testfile in get_files_of_type(files_list, 'test'): 967 fullpath = os.path.join(package_dir, testfile) 968 code = CodeParser(fullpath, self.cheesecake.log.debug) 969 970 functions_tested = functions_tested.union(code.functions_called) 971 972 for name in functions + classes: 973 if name in functions_tested: 974 unittest_cnt += 1 975 self.cheesecake.log.debug("%s is unit tested" % name) 976 977 functions_classes_cnt = len(functions) + len(classes) 978 percent = 0 979 if functions_classes_cnt > 0: 980 percent = float(unittest_cnt)/float(functions_classes_cnt) 981 982 # Scale the result. 983 self.value = int(ceil(percent * self.max_value)) 984 985 self.details = "found %d/%d=%.2f%% unit tested classes/methods/functions." %\ 986 (unittest_cnt, functions_classes_cnt, percent*100) 987 988 return self.value 989 990 class IndexUnitTested(Index): 991 """Check if the package have unit tests which can be easily found by 992 any of known test frameworks. 993 """ 994 max_value = 30 995 996 def compute(self, doctests_count, unittests_count, files_list, classes, methods): 997 unit_tested = False 998 999 if doctests_count > 0: 1000 self.add_info("Package includes doctest tests.") 1001 unit_tested = True 1002 1003 if unittests_count > 0: 1004 self.add_info("Package have tests that inherit from unittest.TestCase.") 1005 unit_tested = True 1006 1007 if get_files_of_type(files_list, 'test'): 1008 self.add_info("Package have filenames which probably contain tests (in format test_* or *_test)") 1009 unit_tested = True 1010 1011 for method in methods: 1012 if self._is_test_method(method): 1013 self.add_info("Some classes have setUp/tearDown methods which are commonly used in unit tests.") 1014 unit_tested = True 1015 break 1016 1017 if unit_tested: 1018 self.value = self.max_value 1019 self.details = "has unit tests" 1020 else: 1021 self.value = 0 1022 self.details = "doesn't have unit tests" 1023 1024 return self.value 1025 1026 def _is_test_method(self, method): 1027 nose_methods = ['setup', 'setup_package', 'setup_module', 'setUp', 1028 'setUpPackage', 'setUpModule', 1029 'teardown', 'teardown_package', 'teardown_module', 1030 'tearDown', 'tearDownModule', 'tearDownPackage'] 1031 1032 for test_method in nose_methods: 1033 if method.endswith(test_method): 1034 return True 1035 return False 1036 1037 class IndexPyLint(Index): 1038 """Compute pylint index of the whole package. 1039 """ 1040 name = "pylint" 1041 max_value = 50 1042 1043 disabled_messages = [ 1044 'W0403', # relative import 1045 'W0406', # importing of self 1046 ] 1047 pylint_args = ' '.join(map(lambda x: '--disable-msg=%s' % x, disabled_messages)) 1048 1049 def compute(self, files_list, package_dir): 1050 # Maximum length of arguments (not very precise). 1051 max_arguments_length = 65536 1052 1053 # Exclude __init__.py files from score as they cause pylint 1054 # to fail with ImportError "Unable to find module for %s in %s". 1055 files_to_lint = filter(lambda name: not name.endswith('__init__.py'), 1056 get_files_of_type(files_list, 'module')) 1057 1058 # Switching cwd so that pylint works correctly regarding 1059 # running it on individual modules. 1060 original_cwd = os.getcwd() 1061 1062 # Note: package_dir may be a file if the archive contains a single file. 1063 # If this is the case, change dir to the parent dir of that file. 1064 if os.path.isfile(package_dir): 1065 package_dir = os.path.dirname(package_dir) 1066 1067 os.chdir(package_dir) 1068 1069 pylint_score = 0 1070 count = 0 1071 error_count = 0 1072 1073 for filenames in generate_arguments(files_to_lint, max_arguments_length - len(self.pylint_args)): 1074 filenames = ' '.join(filenames) 1075 self.cheesecake.log.debug("Running pylint on files: %s." % filenames) 1076 1077 rc, output = run_cmd("pylint %s --persistent=n %s" % (filenames, self.pylint_args)) 1078 if rc: 1079 self.cheesecake.log.debug("encountered an error (%d):\n***\n%s\n***\n" % (rc, output)) 1080 error_count += 1 1081 else: 1082 # Extract score from pylint output. 1083 score_line = output.split("\n")[-3] 1084 s = re.search(r" (-?\d+\.\d+)/10", score_line) 1085 if s: 1086 pylint_score += float(s.group(1)) 1087 count += 1 1088 1089 # Switching back to the original cwd. 1090 os.chdir(original_cwd) 1091 1092 if count: 1093 pylint_score = float(pylint_score)/float(count) 1094 self.details = "pylint score was %.2f out of 10" % pylint_score 1095 elif error_count: 1096 self.details = "encountered an error during pylint execution" 1097 else: 1098 self.details = "no files to check found" 1099 1100 # Assume scores below zero as zero for means of index value computation. 1101 if pylint_score < 0: 1102 pylint_score = 0 1103 self.value = int(ceil(pylint_score/10.0 * self.max_value)) 1104 1105 self.add_info("Score is %.2f/10, which is %d%% of maximum %d points = %d." % 1106 (pylint_score, int(pylint_score*10), self.max_value, self.value)) 1107 1108 return self.value 1109 1110 def decide_before_download(self, cheesecake): 1111 # Try to run the pylint script 1112 if not command_successful("pylint --version"): 1113 cheesecake.log.debug("pylint not properly installed, omitting pylint index.") 1114 return False 1115 1116 return not cheesecake.lite 1117 1118 class IndexCodeKwalitee(Index): 1119 name = "CODE KWALITEE" 1120 1121 subindices = [ 1122 IndexPyLint, 1123 #IndexUnitTests, 1124 IndexUnitTested, 1125 ] 1126 1127 ################################################################################ 1128 ## Main Cheesecake class. 1129 ################################################################################ 98 1130 99 1131 class CheesecakeError(Exception): 100 """ 101 Custom exception class for Cheesecake-specific errors 1132 """Custom exception class for Cheesecake-specific errors. 102 1133 """ 103 1134 pass 104 1135 1136 1137 class CheesecakeIndex(Index): 1138 name = "Cheesecake" 1139 subindices = [ 1140 IndexInstallability, 1141 IndexDocumentation, 1142 IndexCodeKwalitee, 1143 ] 1144 1145 1146 class Step(object): 1147 """Single step during computation of package score. 1148 """ 1149 def __init__(self, provides): 1150 self.provides = provides 1151 1152 def decide(self, cheesecake): 1153 """Decide if step should be run. 1154 1155 It checks if there's at least one index from current profile that need 1156 variables provided by this step. Override this method for other behaviour. 1157 """ 1158 for provide in self.provides: 1159 if provide in cheesecake.index.requirements: 1160 return True 1161 return False 1162 1163 class StepByVariable(Step): 1164 """Step which is always run if given Cheesecake instance variable is true. 1165 """ 1166 def __init__(self, variable_name, provides): 1167 self.variable_name = variable_name 1168 Step.__init__(self, provides) 1169 1170 def decide(self, cheesecake): 1171 if getattr(cheesecake, self.variable_name, None): 1172 return True 1173 1174 # Fallback to the default. 1175 return Step.decide(self, cheesecake) 1176 105 1177 class Cheesecake(object): 106 """ 107 Computes 'goodness' of Python packages 1178 """Computes 'goodness' of Python packages. 108 1179 109 1180 Generates "cheesecake index" that takes into account things like: … … 119 1190 """ 120 1191 121 def __init__(self, name="", url="", path="", sandbox=None, 122 verbose=False, quiet=False): 123 """ 124 Initialize critical variables, download and unpack package, walk package tree 125 1192 steps = {} 1193 1194 package_types = { 1195 "tar.gz": untar_package, 1196 "tgz": untar_package, 1197 "zip": unzip_package, 1198 "egg": unegg_package, 1199 } 1200 1201 def __init__(self, name="", url="", path="", sandbox=None, 1202 logfile=None, verbose=False, quiet=False, static_only=False, 1203 lite=False, keep_log=False): 1204 """Initialize critical variables, download and unpack package, 1205 walk package tree. 126 1206 """ 127 1207 self.name = name 128 1208 self.url = url 129 1209 self.package_path = path 130 if not self.name and not self.url and not self.package_path: 131 self.raise_exception("No package name, URL or path specified ... exiting") 132 self.sandbox = sandbox or "/tmp/cheesecake_sandbox" 1210 1211 if self.name: 1212 self.package = self.name 1213 elif self.url: 1214 self.package = get_package_name_from_url(self.url) 1215 elif self.package_path: 1216 self.package = get_package_name_from_path(self.package_path) 1217 else: 1218 self.raise_exception("No package name, URL or path specified... exiting") 1219 1220 # Setup a sandbox. 1221 self.sandbox = sandbox or tempfile.mkdtemp(prefix='cheesecake') 133 1222 if not os.path.isdir(self.sandbox): 134 1223 os.mkdir(self.sandbox) 1224 135 1225 self.verbose = verbose 136 1226 self.quiet = quiet 137 138 self.package_types = ["tar.gz", "tgz", "zip"] 1227 self.static_only = static_only 1228 self.lite = lite 1229 self.keep_log = keep_log 1230 139 1231 self.sandbox_pkg_file = "" 140 1232 self.sandbox_pkg_dir = "" 141 1233 self.sandbox_install_dir = "" 142 1234 143 self.determine_pkg_name() 144 self.configure_logging() 145 self.set_defaults() 146 self.get_config() 147 self.init_indexes() 148 self.retrieve_pkg() 149 self.unpack_pkg() 150 self.walk_pkg() 1235 # Configure logging as soon as possible. 1236 self.configure_logging(logfile) 1237 1238 # Setup Cheesecake index. 1239 self.index = CheesecakeIndex() 1240 1241 self.index.decide_before_download(self) 1242 self.log.debug("Profile requirements: %s." % ', '.join(sorted(self.index.requirements))) 1243 1244 # Get the package. 1245 self.run_step('get_pkg_from_pypi') 1246 self.run_step('download_pkg') 1247 self.run_step('copy_pkg') 1248 1249 # Get package name and type. 1250 name_and_type = get_package_name_and_type(self.package, self.package_types.keys()) 1251 1252 if not name_and_type: 1253 msg = "Could not determine package type for package '%s'" % self.package 1254 msg += "\nCurrently recognized types: " + ", ".join(self.package_types.keys()) 1255 self.raise_exception(msg) 1256 1257 self.package_name, self.package_type = name_and_type 1258 self.log.debug("Package name: " + self.package_name) 1259 self.log.debug("Package type: " + self.package_type) 1260 1261 # Make last indices decisions. 1262 self.index.decide_after_download(self) 1263 1264 # Unpack package and list its files. 1265 self.run_step('unpack_pkg') 1266 self.run_step('walk_pkg') 1267 1268 # Install package. 1269 self.run_step('install_pkg') 151 1270 152 1271 def raise_exception(self, msg): 153 """ 154 Cleanup, print error message and raise CheesecakeError 155 156 Don't use logging, since it can be called before logging has been setup 157 """ 158 self.cleanup() 159 os.unlink(os.path.join(self.sandbox, self.logfile)) 160 161 msg += "\n" + pad_msg("CHEESECAKE INDEX", 0) 162 raise CheesecakeError(msg) 163 164 def cleanup(self): 165 """ 166 Delete temporary directories and files that were 167 created in the sandbox 1272 """Cleanup, print error message and raise CheesecakeError. 1273 1274 Don't use logging, since it can be called before logging has been setup. 1275 """ 1276 self.cleanup(remove_log_file=False) 1277 1278 msg += "\nDetailed info available in log file %s" % self.logfile 1279 1280 raise CheesecakeError("Error: " + msg) 1281 1282 def cleanup(self, remove_log_file=True): 1283 """Delete temporary directories and files that were created 1284 in the sandbox. At the end delete the sandbox itself. 168 1285 """ 169 1286 if os.path.isfile(self.sandbox_pkg_file): 170 1287 self.log("Removing file %s" % self.sandbox_pkg_file) 171 1288 os.unlink(self.sandbox_pkg_file) 172 if os.path.isdir(self.sandbox_pkg_dir): 173 self.log("Removing directory %s" % self.sandbox_pkg_dir) 174 shutil.rmtree(self.sandbox_pkg_dir) 175 if os.path.isdir(self.sandbox_install_dir): 176 self.log("Removing directory %s" % self.sandbox_install_dir) 177 shutil.rmtree(self.sandbox_install_dir) 178 179 def set_defaults(self): 180 """ 181 Set default values for variables that can also be defined 182 in the config file 183 """ 184 self.INDEX_PYPI_DOWNLOAD = 50 185 self.INDEX_PYPI_DISTANCE = 5 186 self.INDEX_URL_DOWNLOAD = 25 187 self.INDEX_UNPACK = 25 188 self.INDEX_UNPACK_DIR = 15 189 self.INDEX_INSTALL = 50 190 self.INDEX_FILE_CRITICAL = 15 191 self.INDEX_FILE = 10 192 self.INDEX_REQUIRED_FILES = 100 193 self.INDEX_FILE_PYC = -20 194 self.INDEX_DIR_CRITICAL = 25 195 self.INDEX_DIR = 20 196 self.INDEX_DIR_EMPTY = 5 197 self.MAX_INDEX_DOCSTRINGS = 100 # max. percentage of modules/classes/methods/functions with docstrings 198 self.MAX_INDEX_UNITTESTS = 100 # max. percentage of methods/functions that are unit tested 199 self.MAX_INDEX_PYLINT = 100 # max. pylint score 200 self.cheese_files = ["readme", "install", "changelog", 201 "news", "faq", 202 "todo", "thanks", 203 "license", "announce", 204 "setup.py", 205 ] 206 self.critical_cheese_files = ["readme", "license", "setup.py"] 207 self.cheese_dirs = ["doc", "test", "example", "demo"] 208 self.critical_cheese_dirs = ["doc", "test"] 209 210 def get_config(self): 211 """ 212 Retrieve values from configuration file 213 """ 214 self.config = get_pkg_config(self.short_pkg_name) 215 for config_var in ["INDEX_PYPI_DOWNLOAD", "INDEX_PYPI_DISTANCE", 216 "INDEX_URL_DOWNLOAD", "INDEX_UNPACK", "INDEX_UNPACK_DIR", 217 "INDEX_INSTALL", "INDEX_FILE_CRITICAL", "INDEX_FILE", 218 "INDEX_REQUIRED_FILES", "INDEX_FILE_PYC", 219 "INDEX_DIR_CRITICAL", "INDEX_DIR", "INDEX_DIR_EMPTY", 220 "MAX_INDEX_DOCSTRINGS", "MAX_INDEX_PYLINT", 221 "cheese_files", "critical_cheese_files", 222 "cheese_dirs", "critical_cheese_dirs", 223 ]: 224 value = self.config.get(config_var) 225 if value: setattr(self, config_var, value) 226 227 def determine_pkg_name(self): 228 if self.name: 229 self.package = self.name 230 self.short_pkg_name = self.name 231 elif self.package_path: 232 self.package = self.get_package_from_path(self.package_path) 1289 1290 def delete_dir(dirname): 1291 "Delete directory recursively and generate log message." 1292 if os.path.isdir(dirname): 1293 self.log("Removing directory %s" % dirname) 1294 shutil.rmtree(dirname) 1295 1296 delete_dir(self.sandbox) 1297 1298 if remove_log_file and not self.keep_log: 1299 os.unlink(os.path.join(self.sandbox, self.logfile)) 1300 1301 def configure_logging(self, logfile=None): 1302 """Default settings for logging. 1303 1304 If verbose, log goes to console, else it goes to logfile. 1305 log.debug and log.info goes to logfile. 1306 log.warn and log.error go to both logfile and stdout. 1307 """ 1308 if logfile: 1309 self.logfile = logfile 233 1310 else: 234 self.package = self.get_package_from_url() 235 236 def get_package_from_url(self): 237 """ 238 Use ``urlparse`` to obtain package path from URL 239 """ 240 (scheme,location,path,param,query,fragment_id) = urlparse(self.url) 241 return self.get_package_from_path(path) 242 243 244 def get_package_from_path(self, path): 245 """ 246 Get package name as file portion of path 247 """ 248 dir, file = os.path.split(path) 249 self.short_pkg_name = file 250 for package_type in self.package_types: 251 s = re.search("(.+)\.%s" % package_type, file) 252 if s: 253 self.short_pkg_name = s.group(1) 254 break 255 return file 256 257 def configure_logging(self): 258 """ 259 Default settings for logging 260 261 if verbose, log goes to console, else it goes to logfile 262 log.debug goes to logfile 263 log.info goes to console 264 log.warn and log.error go to both logfile and stdout 265 """ 266 self.logfile = os.path.join(self.sandbox, self.short_pkg_name + ".log") 1311 self.logfile = os.path.join(tempfile.gettempdir(), self.package + ".log") 267 1312 268 1313 logger.setconsumer('logfile', open(str(self.logfile), 'w', buffering=1)) … … 270 1315 logger.setconsumer('null', None) 271 1316 272 if self.verbose: 273 self.log = logger.MultipleProducer('cheesecake console') 274 else: 275 self.log = logger.MultipleProducer('cheesecake logfile') 276 if self.quiet: 277 self.log.info = logger.MultipleProducer('cheesecake logfile') 278 else: 279 self.log.info = logger.MultipleProducer('cheesecake console') 1317 self.log = logger.MultipleProducer('cheesecake logfile') 1318 self.log.info = logger.MultipleProducer('cheesecake logfile') 280 1319 self.log.debug = logger.MultipleProducer('cheesecake logfile') 281 1320 self.log.warn = logger.MultipleProducer('cheesecake console') 282 1321 self.log.error = logger.MultipleProducer('cheesecake console') 283 1322 284 self.log.debug("package = ", self.short_pkg_name) 285 286 def init_indexes(self): 287 """ 288 Initialize variables used in index computation 289 290 * cheesecake_index: overall index for the package 291 * index: dict holding Index or CompositeIndex objects of various types 292 """ 293 self.cheesecake_index = 0 294 self.cheesecake_index_installability = 0 295 self.cheesecake_index_documentation = 0 296 self.cheesecake_index_codekwalitee = 0 297 self.max_cheesecake_index = self.INDEX_PYPI_DOWNLOAD + \ 298 self.INDEX_UNPACK + \ 299 self.INDEX_UNPACK_DIR + \ 300 self.INDEX_INSTALL + \ 301 self.MAX_INDEX_DOCSTRINGS + \ 302 self.MAX_INDEX_PYLINT 303 # self.MAX_INDEX_UNITTESTS 304 self.max_cheesecake_index_installability = self.INDEX_PYPI_DOWNLOAD + \ 305 self.INDEX_UNPACK + \ 306 self.INDEX_UNPACK_DIR + \ 307 self.INDEX_INSTALL 308 self.max_cheesecake_index_documentation = self.INDEX_REQUIRED_FILES + \ 309 self.MAX_INDEX_DOCSTRINGS 310 self.max_cheesecake_index_codekwalitee = self.MAX_INDEX_PYLINT 311 # self.MAX_INDEX_UNITTESTS 312 313 self.index = {} 314 for index_type in ["file", "dir"]: 315 self.index[index_type] = CompositeIndex(index_type) 316 for index_type in ["pypi_download", "url_download", 317 "unpack_dir", "unpack", "install", 318 "docstrings", "unittests", "pylint"]: 319 self.index[index_type] = Index(index_type) 320 321 for cheese_file in self.cheese_files: 322 self.index["file"].set_index(name=cheese_file, details="file not found") 323 if cheese_file in self.critical_cheese_files: 324 self.max_cheesecake_index += self.INDEX_FILE_CRITICAL 325 self.max_cheesecake_index_documentation += self.INDEX_FILE_CRITICAL 326 else: 327 self.max_cheesecake_index += self.INDEX_FILE 328 self.max_cheesecake_index_documentation += self.INDEX_FILE 329 self.log.debug("cheese_files: " + ",".join(self.cheese_files)) 330 self.log.debug("critical_cheese_files: " + ",".join(self.critical_cheese_files)) 331 332 for cheese_dir in self.cheese_dirs: 333 self.index["dir"].set_index(name=cheese_dir, details="directory not found") 334 if cheese_dir in self.critical_cheese_dirs: 335 self.max_cheesecake_index += self.INDEX_DIR_CRITICAL 336 self.max_cheesecake_index_documentation += self.INDEX_DIR_CRITICAL 337 else: 338 self.max_cheesecake_index += self.INDEX_DIR 339 self.max_cheesecake_index_documentation += self.INDEX_DIR 340 self.log.debug("cheese_dirs: " + ",".join(self.cheese_dirs)) 341 self.log.debug("critical_cheese_dirs: " + ",".join(self.critical_cheese_dirs)) 342 343 self.pkg_files = {} 344 self.pkg_dirs = {} 345 self.file_types = ["py", "pyc", "test", 346 ] 347 for type in self.file_types: 348 self.pkg_files[type] = [] 349 350 self.object_cnt = 0 # Number of modules/functions/classes/methods in .py files found 351 self.docstring_cnt = 0 352 self.functions = [] # List of methods/functions found in .py files 353 354 def retrieve_pkg(self): 355 if self.name: 356 self.get_pkg_from_pypi() 357 elif self.url: 358 self.download_pkg() 359 else: 360 self.copy_pkg() 361 362 def get_package_from_url(self): 363 """ 364 Use ``urlparse`` to obtain package path from URL 365 """ 366 (scheme,location,path,param,query,fragment_id) = urlparse(self.url) 367 return self.get_package_from_path(path) 368 369 370 def get_package_from_path(self, path): 371 """ 372 Get package name as file portion of path 373 """ 374 dir, file = os.path.split(path) 375 self.short_pkg_name = file 376 for package_type in self.package_types: 377 s = re.search("(.+)\.%s" % package_type, file) 378 if s: 379 self.short_pkg_name = s.group(1) 380 break 381 return file 382 1323 def run_step(self, step_name): 1324 """Run step if its decide() method returns True. 1325 """ 1326 step = self.steps[step_name] 1327 if step.decide(self): 1328 step_method = getattr(self, step_name) 1329 step_method() 1330 1331 steps['get_pkg_from_pypi'] = StepByVariable('name', 1332 ['download_url', 1333 'distance_from_pypi', 1334 'found_on_cheeseshop', 1335 'found_locally', 1336 'sandbox_pkg_file']) 383 1337 def get_pkg_from_pypi(self): 384 """ 385 Download package using setuptools utilities 386 """ 1338 """Download package using setuptools utilities. 1339 1340 New attributes: 1341 download_url : str 1342 URL that package was downloaded from. 1343 distance_from_pypi : int 1344 How many hops setuptools had to make to download package. 1345 found_on_cheeseshop : bool 1346 Whenever package has been found on CheeseShop. 1347 found_locally : bool 1348 Whenever package has been already installed. 1349 """ 1350 self.log.info("Trying to download package %s from PyPI using setuptools utilities" % self.name) 1351 387 1352 try: 388 self.log.info("Trying to download package %s from PyPI using setuptools utilities" % self.name)389 1353 from setuptools.package_index import PackageIndex 390 1354 from pkg_resources import Requirement 391 1355 from distutils import log 392 # Temporarily set the log verbosity to INFO so we can capture setuptools info messages 393 old_threshold = log.set_threshold(log.INFO) 394 pkgindex = PackageIndex() 395 old_stdout = sys.stdout 396 sys.stdout = StdoutRedirector() 397 output = pkgindex.fetch(Requirement.parse(self.name), 398 self.sandbox, 399 force_scan=True, 400 source=True) 401 captured_stdout = sys.stdout.read_buffer() 402 sys.stdout = old_stdout 403 log.set_threshold(old_threshold) 404 if output is None: 405 self.raise_exception("Error: Could not find distribution for " + self.name) 406 download_url = "" 407 distance_from_pypi = 0 408 #print captured_stdout 409 for line in captured_stdout.split('\n'): 410 s = re.search(r"Reading http(.*)", line) 411 if s: 412 inspected_url = s.group(1) 413 if not re.search(r"www.python.org\/pypi", inspected_url): 414 distance_from_pypi += 1 415 continue 416 s = re.search(r"Downloading (.*)", line) 417 if s: 418 download_url = s.group(1) 419 break 420 self.sandbox_pkg_file = output 421 self.package = self.get_package_from_path(output) 422 self.log.info("Downloaded package %s from %s" % (self.package, download_url)) 423 index_type = "pypi_download" 424 found_on_cheeseshop = False 425 if re.search(r"cheeseshop.python.org", download_url): 426 value = self.INDEX_PYPI_DOWNLOAD 427 found_on_cheeseshop = True 428 else: 429 value = self.INDEX_PYPI_DOWNLOAD - distance_from_pypi * self.INDEX_PYPI_DISTANCE 430 self.index[index_type].value = value 431 details = "downloaded package " + self.package 432 if found_on_cheeseshop: 433 details += " directly from the Cheese Shop" 434 elif distance_from_pypi: 435 details += " following %d link" % distance_from_pypi 436 if distance_from_pypi > 1: 437 details += "s" 438 details += " from PyPI" 439 else: 440 details += "from " + download_url 441 self.index[index_type].details = details 1356 from distutils.errors import DistutilsError 442 1357 except ImportError, e: 443 msg = " Error:setuptools is not installed and is required for downloading a package by name\n"444 msg += "You can do nwload and process a package by its full URL via the -u or --url option\n"1358 msg = "setuptools is not installed and is required for downloading a package by name\n" 1359 msg += "You can download and process a package by its full URL via the -u or --url option\n" 445 1360 msg += "Example: python cheesecake.py --url=http://www.mems-exchange.org/software/durus/Durus-3.1.tar.gz" 446 1361 self.raise_exception(msg) 447 1362 1363 def drop_setuptools_info(stdout, error=None): 1364 """Drop all setuptools output as INFO. 1365 """ 1366 self.log.info("*** Begin setuptools output") 1367 map(self.log.info, stdout.splitlines()) 1368 if error: 1369 self.log.info(str(error)) 1370 self.log.info("*** End setuptools output") 1371 1372 def fetch_package(mode): 1373 """Fetch package from PyPI. 1374 1375 Mode can be one of: 1376 * 'pypi_source': get source package from PyPI 1377 * 'pypi_any': get source/egg package from PyPI 1378 * 'any': get package from PyPI or local filesystem 1379 1380 Returns tuple (status, output), where `status` is True 1381 if fetch was successful and False if it failed. `output` 1382 is PackageIndex.fetch() return value. 1383 """ 1384 if 'pypi' in mode: 1385 pkgindex = PackageIndex(search_path=[]) 1386 else: 1387 pkgindex = PackageIndex() 1388 1389 if mode == 'pypi_source': 1390 source = True 1391 else: 1392 source = False 1393 1394 try: 1395 output = pkgindex.fetch(Requirement.parse(self.name), 1396 self.sandbox, 1397 force_scan=True, 1398 source=source) 1399 return True, output 1400 except DistutilsError, e: 1401 return False, e 1402 1403 # Temporarily set the log verbosity to INFO so we can capture setuptools 1404 # info messages. 1405 old_threshold = log.set_threshold(log.INFO) 1406 old_stdout = sys.stdout 1407 sys.stdout = StdoutRedirector() 1408 1409 # Try to get source package from PyPI first, then egg from PyPI, and if 1410 # that fails search in locally installed packages. 1411 for mode, info in [('pypi_source', "source package on PyPI"), 1412 ('pypi_any', "egg on PyPI"), 1413 ('any', "locally installed package")]: 1414 msg = "Looking for %s... " % info 1415 status, output = fetch_package(mode) 1416 if status and output: 1417 self.log.info(msg + "found!") 1418 break 1419 self.log.info(msg + "failed.") 1420 1421 # Bring back old stdout. 1422 captured_stdout = sys.stdout.read_buffer() 1423 sys.stdout = old_stdout 1424 log.set_threshold(old_threshold) 1425 1426 # If all runs failed, we must raise an error. 1427 if not status: 1428 drop_setuptools_info(captured_stdout, output) 1429 self.raise_exception("setuptools returned an error: %s\n" % str(output).splitlines()[0]) 1430 1431 # If fetch returned nothing, package wasn't found. 1432 if output is None: 1433 drop_setuptools_info(captured_stdout) 1434 self.raise_exception("Could not find distribution for " + self.name) 1435 1436 # Defaults. 1437 self.download_url = "" 1438 self.distance_from_pypi = 0 1439 self.found_on_cheeseshop = False 1440 self.found_locally = False 1441 1442 for line in captured_stdout.splitlines(): 1443 s = re.search(r"Reading http(.*)", line) 1444 if s: 1445 inspected_url = s.group(1) 1446 if not re.search(r"www.python.org\/pypi", inspected_url): 1447 self.distance_from_pypi += 1 1448 continue 1449 s = re.search(r"Downloading (.*)", line) 1450 if s: 1451 self.download_url = s.group(1) 1452 break 1453 1454 self.sandbox_pkg_file = output 1455 self.package = get_package_name_from_path(output) 1456 self.log.info("Downloaded package %s from %s" % (self.package, self.download_url)) 1457 1458 if os.path.isdir(self.sandbox_pkg_file): 1459 self.found_locally = True 1460 1461 if re.search(r"cheeseshop.python.org", self.download_url): 1462 self.found_on_cheeseshop = True 1463 1464 steps['download_pkg'] = StepByVariable('url', 1465 ['sandbox_pkg_file', 1466 'downloaded_from_url']) 448 1467 def download_pkg(self): 449 """ 450 Use ``urllib.urlretrieve`` to download package to file in sandbox dir 1468 """Use ``urllib.urlretrieve`` to download package to file in sandbox dir. 451 1469 """ 452 1470 #self.log("Downloading package %s from URL %s" % (self.package, self.url)) … … 458 1476 self.raise_exception(str(e)) 459 1477 #self.log("Downloaded package %s to %s" % (self.package, downloaded_filename)) 460 if re.search("Content-Type: details/html", str(headers)): 1478 1479 if headers.gettype() in ["text/html"]: 461 1480 f = open(downloaded_filename) 462 1481 if re.search("404 Not Found", "".join(f.readlines())): … … 464 1483 self.raise_exception("Got '404 Not Found' error while trying to download package ... exiting") 465 1484 f.close() 466 index_type = "url_download" 467 self.index[index_type].value = self.INDEX_URL_DOWNLOAD 468 self.index[index_type].details = "downloaded package %s from URL %s" % (self.package, self.url) 469 1485 1486 self.downloaded_from_url = True 1487 1488 steps['copy_pkg'] = StepByVariable('package_path', 1489 ['sandbox_pkg_file']) 470 1490 def copy_pkg(self): 471 """ 472 Copy package file to sandbox directory 1491 """Copy package file to sandbox directory. 473 1492 """ 474 1493 self.sandbox_pkg_file = os.path.join(self.sandbox, self.package) … … 478 1497 shutil.copyfile(self.package_path, self.sandbox_pkg_file) 479 1498 1499 steps['unpack_pkg'] = Step(['original_package_name', 1500 'sandbox_pkg_dir', 1501 'unpacked', 1502 'unpack_dir']) 480 1503 def unpack_pkg(self): 481 """ 482 Unpack the package in the sandbox directory 483 484 Currently supported archive types: 485 486 * .tar.gz (handled with ``tarfile`` module) 487 * .zip (handled with ``zipfile`` module) 488 """ 489 self.package_type = "" 490 for type in self.package_types: 491 s = re.search(r"(.+)\.%s" % type, self.package) 492 if s: 493 # package_name is name of package without file extension (ex. twill-7.3) 494 self.package_name = s.group(1) 495 self.package_type = type 496 break 497 if not self.package_type: 498 msg = "Could not determine package type for package '%s'" % self.package 499 msg += "\nCurrently recognized types: " + " ".join(self.package_types) 500 self.raise_exception(msg) 501 self.log.debug("Package name: " + self.package_name) 502 self.log.debug("Package type: " + self.package_type) 503 1504 """Unpack the package in the sandbox directory. 1505 1506 Check `package_types` attribute for list of currently supported 1507 archive types. 1508 1509 New attributes: 1510 original_package_name : str 1511 Package name guessed from the package name. Will be set only 1512 if package name is different than unpacked directory name. 1513 """ 504 1514 self.sandbox_pkg_dir = os.path.join(self.sandbox, self.package_name) 505 1515 if os.path.isdir(self.sandbox_pkg_dir): 506 1516 shutil.rmtree(self.sandbox_pkg_dir) 507 1517 508 if self.package_type in ["tar.gz", "tgz"]: 509 self.untar_pkg() 510 elif self.package_type == "zip": 511 self.unzip_pkg() 512 513 index_type = "unpack_dir" 514 details = "unpack directory is " + self.unpack_dir 1518 # Call appropriate function to unpack the package. 1519 unpack = self.package_types[self.package_type] 1520 self.unpack_dir = unpack(self.sandbox_pkg_file, self.sandbox) 1521 1522 if self.unpack_dir is None: 1523 self.raise_exception("Could not unpack package %s ... exiting" % \ 1524 self.sandbox_pkg_file) 1525 1526 self.unpacked = True 1527 515 1528 if self.unpack_dir != self.package_name: 516 details += " instead of the expected " +self.package_name1529 self.original_package_name = self.package_name 517 1530 self.package_name = self.unpack_dir 1531 1532 steps['walk_pkg'] = Step(['dirs_list', 1533 'docstring_cnt', 1534 'docformat_cnt', 1535 'doctests_count', 1536 'unittests_count', 1537 'files_list', 1538 'functions', 1539 'classes', 1540 'methods', 1541 'object_cnt', 1542 'package_dir']) 1543 def walk_pkg(self): 1544 """Get package files and directories. 1545 1546 New attributes: 1547 dirs_list : list 1548 List of directories package contains. 1549 docstring_cnt : int 1550 Number of docstrings found in all package objects. 1551 docformat_cnt : int 1552 Number of formatted docstrings found in all package objects. 1553 doctests_count : int 1554 Number of docstrings that include doctests. 1555 unittests_count : int 1556 Number of classes which inherit from unittest.TestCase. 1557 files_list : list 1558 List of files package contains. 1559 functions : list 1560 List of all functions defined in package sources. 1561 classes : list 1562 List of all classes defined in package sources. 1563 methods : list 1564 List of all methods defined in package sources. 1565 object_cnt : int 1566 Number of documentable objects found in all package modules. 1567 package_dir : str 1568 Path to project directory. 1569 """ 1570 self.package_dir = os.path.join(self.sandbox, self.package_name) 1571 1572 self.files_list, self.dirs_list = get_files_dirs_list(self.package_dir) 1573 1574 self.object_cnt = 0 1575 self.docstring_cnt = 0 1576 self.docformat_cnt = 0 1577 self.doctests_count = 0 1578 self.functions = [] 1579 self.classes = [] 1580 self.methods = [] 1581 self.unittests_count = 0 1582 1583 # Parse all application files and count objects 1584 # (modules/classes/functions) and their associated docstrings. 1585 for py_file in get_files_of_type(self.files_list, 'module'): 1586 pyfile = os.path.join(self.package_dir, py_file) 1587 code = CodeParser(pyfile, self.log.debug) 1588 1589 self.object_cnt += code.object_count() 1590 self.docstring_cnt += code.docstring_count() 1591 self.docformat_cnt += code.formatted_docstrings_count 1592 self.functions += code.functions 1593 self.classes += code.classes 1594 self.methods += code.methods 1595 self.doctests_count += code.doctests_count 1596 self.unittests_count += code.unittests_count 1597 1598 # Log a bit of debugging info. 1599 self.log.debug("Found %d files: %s." % (len(self.files_list), 1600 ', '.join(self.files_list))) 1601 self.log.debug("Found %d directories: %s." % (len(self.dirs_list), 1602 ', '.join(self.dirs_list))) 1603 1604 steps['install_pkg'] = Step(['installed']) 1605 def install_pkg(self): 1606 """Verify that package can be installed in alternate directory. 1607 1608 New attributes: 1609 installed : bool 1610 Describes whenever package has been succefully installed. 1611 """ 1612 self.sandbox_install_dir = os.path.join(self.sandbox, "tmp_install_%s" % self.package_name) 1613 1614 if self.package_type == 'egg': 1615 # Create dummy Python directories. 1616 mkdirs('%s/lib/python2.3/site-packages/' % self.sandbox_install_dir) 1617 mkdirs('%s/lib/python2.4/site-packages/' % self.sandbox_install_dir) 1618 1619 environment = {'PYTHONPATH': 1620 '%(sandbox)s/lib/python2.3/site-packages/:'\ 1621 '%(sandbox)s/lib/python2.4/site-packages/' % \ 1622 {'sandbox': self.sandbox_install_dir}, 1623 # Pass PATH to child process. 1624 'PATH': os.getenv('PATH')} 1625 rc, output = run_cmd("easy_install --no-deps --prefix %s %s" % \ 1626 (self.sandbox_install_dir, 1627 self.sandbox_pkg_file), 1628 environment) 518 1629 else: 519 details += " as expected" 520 self.index[index_type].value = self.INDEX_UNPACK_DIR 521 self.index[index_type].details = details 522 523 if not self.quiet: 524 self.log.info("Detailed info available in log file %s" % self.logfile) 525 526 def untar_pkg(self): 527 """ 528 Untar the package in the sandbox directory 529 530 Uses tarfile module 531 """ 532 try: 533 t = tarfile.open(self.sandbox_pkg_file) 534 except tarfile.ReadError, e: 535 self.raise_exception("Could not read tar file %s ... exiting" % self.sandbox_pkg_file) 536 537 for member in t.getmembers(): 538 t.extract(member, self.sandbox) 539 540 tarinfo = t.members[0] 541 self.unpack_dir = tarinfo.name.split(os.sep)[0] 542 543 index_type = "unpack" 544 self.index[index_type].value = self.INDEX_UNPACK 545 self.index[index_type].details = "package untar-ed successfully" 546 547 def unzip_pkg(self): 548 """ 549 Unzip the package in the sandbox directory 550 551 Uses zipfile module 552 """ 553 try: 554 z = zipfile.ZipFile(self.sandbox_pkg_file) 555 except zipfile.error: 556 self.raise_exception("Error unzipping file %s ... exiting" % self.sandbox_pkg_file) 557 558 # Get directory structure from zip and create it in sandbox 559 for name in z.namelist(): 560 (dir, file) = os.path.split(name) 561 unpack_dir = dir 562 target_dir = os.path.join(self.sandbox, dir) 563 if not os.path.exists(target_dir): 564 os.makedirs(target_dir) 565 566 # Extract files to directory structure 567 for i, name in enumerate(z.namelist()): 568 if not name.endswith('/'): 569 outfile = open(os.path.join(self.sandbox, name), 'wb') 570 outfile.write(z.read(name)) 571 outfile.flush() 572 outfile.close() 573 574 self.unpack_dir = unpack_dir.split(os.sep)[0] 575 576 index_type = "unpack" 577 self.index[index_type].value = self.INDEX_UNPACK 578 self.index[index_type].details = "package unzipped successfully" 579 580 def walk_pkg(self): 581 """ 582 Traverse the file system tree rooted at sandbox/package_name 583 584 * Compute indexes for special files and directories 585 * Identify Python files, test files, etc. 586 """ 587 cwd = os.getcwd() 588 os.chdir(self.sandbox) 589 for rootdir, dirs, files in os.walk(self.package_name): 590 head, tail = os.path.split(rootdir) 591 dirs_in_rootdir = rootdir.split(os.path.sep) 592 for cheese_dir in self.cheese_dirs: 593 if re.search("^%s" % cheese_dir, tail): 594 if files or dirs: 595 if cheese_dir in self.critical_cheese_dirs: 596 value = self.INDEX_DIR_CRITICAL 597 details = "critical directory found" 598 self.log.debug("critical_cheese_dir found: " + cheese_dir) 599 else: 600 value = self.INDEX_DIR 601 details = "directory found" 602 self.log.debug("cheese_dir found: " + cheese_dir) 603 else: 604 value = self.INDEX_DIR_EMPTY 605 details = "empty directory found" 606 self.log.debug("empty cheese_dir found: " + cheese_dir) 607 self.index["dir"].set_index(cheese_dir, value, details) 608 for file in files: 609 fullpath = os.path.join(rootdir, file) 610 for cheese_file in self.cheese_files: 611 if re.search(r"^%s(\.txt)*" % cheese_file, file, re.IGNORECASE): 612 if cheese_file in self.critical_cheese_files: 613 value = self.INDEX_FILE_CRITICAL 614 details = "critical file found" 615 self.log.debug("critical_cheese_file found: " + cheese_file) 616 else: 617 value = self.INDEX_FILE 618 details = "file found" 619 self.log.debug("cheese_file found: " + cheese_file) 620 self.index["file"].set_index(cheese_file, value, details) 621 622 if self.is_py_file(file, dirs_in_rootdir): 623 self.pkg_files["py"].append(fullpath) 624 self.log.debug("py file found: " + fullpath) 625 pyfile = os.path.join(self.sandbox, fullpath) 626 # Parse the file and count objects (modules/classes/functions) 627 # and their associated docstrings 628 code = CodeParser(pyfile, self.log.debug) 629 self.object_cnt += code.object_count() 630 self.docstring_cnt += code.docstring_count() 631 self.functions += code.functions 632 633 if os.path.splitext(file)[1] == ".pyc": 634 self.pkg_files["pyc"].append(fullpath) 635 self.log.debug("pyc file found: " + fullpath) 636 637 if self.is_test_file(file, dirs_in_rootdir): 638 self.pkg_files["test"].append(fullpath) 639 self.log.debug("test file found: " + fullpath) 640 641 len_pyc_list = len(self.pkg_files["pyc"]) 642 if len_pyc_list: 643 self.index["file"].set_index("pyc", value=self.INDEX_FILE_PYC, 644 details="%d .pyc files found" % len_pyc_list) 645 self.log.debug("Found %d py files" % len(self.pkg_files["py"])) 646 self.log.debug("Found %d pyc files" % len(self.pkg_files["pyc"])) 647 self.log.debug("Found %d test files" % len(self.pkg_files["test"])) 648 649 os.chdir(cwd) 650 651 def is_py_file(self, file, dirs): 652 """ 653 Return True if file ends with .py and it is not a special file and it is not 654 in special directory 655 """ 656 if os.path.splitext(file)[1] != ".py": 657 return False 658 if file in ["setup.py", "ez_setup.py", "__init__.py", "__pkginfo__.py"]: 659 return False 660 for dir in dirs: 661 if dir.startswith("test") or \ 662 dir.startswith("docs") or \ 663 dir.startswith("demo") or \ 664 dir.startswith("example"): 665 return False 666 return True 667 668 def is_test_file(self, file, dirs): 669 """ 670 Return True is file is in directory rooted at "test" or "tests" 671 """ 672 if not file.endswith(".py"): 673 return False 674 if file in ["__init__.py"]: 675 return False 676 for dir in dirs: 677 if dir.startswith("test"): 678 return True 679 return False 680 681 def index_file(self): 682 """ 683 Return CompositeIndex object of type "file" 684 """ 685 return self.index["file"] 686 687 def index_dir(self): 688 """ 689 Return CompositeIndex object of type "dir" 690 """ 691 return self.index["dir"] 692 693 def index_pypi_download(self): 694 """ 695 Verify that package can be downloaded from PyPI 696 697 Return Index object of type "pypi_download" 698 """ 699 index_type = "pypi_download" 700 if self.url: 701 # Package was downloaded directly from URL 702 self.index[index_type].value = 0 703 self.index[index_type].details = "package was downloaded directly from URL" 704 705 if self.package_path: 706 # Package was processed from file system path 707 self.index[index_type].value = 0 708 self.index[index_type].details = "package was processed from file system path" 709 710 # Otherwise, index["pypi_download"] was already set in get_pkg_from_pypi() 711 return self.index["pypi_download"] 712 713 def index_url_download(self): 714 """ 715 Verify that package can be downloaded from an URL 716 717 Return Index object of type "download" 718 """ 719 # index["download"] is already set in download_pkg() 720 return self.index["url_download"] 721 722 def index_unpack(self): 723 """ 724 Verify that package can be unpacked 725 726 Return Index object of type "unpack" 727 """ 728 # index["unpack"] is already set in unpack_pkg() 729 return self.index["unpack"] 730 731 732 def index_unpack_dir(self): 733 """ 734 Verify that unpack directory has same name as package 735 736 Return Index object of type "unpack_dir" 737 """ 738 # index["unpack_dir"] is already set in unpack_pkg() 739 return self.index["unpack_dir"] 740 741 def index_install(self): 742 """ 743 Verify that package can be installed in alternate directory 744 745 Return Index object of type "install" 746 """ 747 index_type = "install" 748 self.sandbox_install_dir = os.path.join(self.sandbox, "tmp_install_%s" % self.package_name) 749 cwd = os.getcwd() 750 os.chdir(os.path.join(self.sandbox, self.package_name)) 751 rc, output = run_cmd("python setup.py install --root=" + self.sandbox_install_dir) 752 if not rc: 753 # Install succeeded 754 self.index[index_type].value = self.INDEX_INSTALL 755 self.index[index_type].details = details="package installed in %s" % self.sandbox_install_dir 1630 package_dir = os.path.join(self.sandbox, self.package_name) 1631 if not os.path.isdir(package_dir): 1632 package_dir = self.sandbox 1633 cwd = os.getcwd() 1634 os.chdir(package_dir) 1635 rc, output = run_cmd("python setup.py install --root=%s" % \ 1636 self.sandbox_install_dir) 1637 os.chdir(cwd) 1638 1639 if rc: 1640 self.log('*** Installation failed. Captured output:') 1641 # Stringify output as it may be an exception. 1642 for output_line in str(output).splitlines(): 1643 self.log(output_line) 1644 self.log('*** End of captured output.') 756 1645 else: 757 # Install failed 758 self.index[index_type].details = "could not install package in %s" % self.sandbox_install_dir 759 os.chdir(cwd) 760 return self.index[index_type] 761 762 def index_docstrings(self): 763 """ 764 Compute docstring index as percentage of modules/classes/methods/functions 765 that have docstrings associated with them 766 767 Return Index object of type "docstrings" 768 """ 769 index_type = "docstrings" 770 if self.object_cnt: 771 percent = float(self.docstring_cnt)/float(self.object_cnt) 1646 self.log('Installation into %s successful.' % \ 1647 self.sandbox_install_dir) 1648 self.installed = True 1649 1650 def compute_cheesecake_index(self): 1651 """Compute overall Cheesecake index for the package by adding up 1652 specific indexes. 1653 """ 1654 # Recursively compute all indices. 1655 max_cheesecake_index = self.index.max_value 1656 1657 # Pass Cheesecake instance to the main Index object. 1658 cheesecake_index = self.index.compute_with(self) 1659 percentage = (cheesecake_index * 100) / max_cheesecake_index 1660 1661 self.log.info("A given package can currently reach a MAXIMUM number of %d points" % max_cheesecake_index) 1662 self.log.info("Starting computation of Cheesecake index for package '%s'" % (self.package)) 1663 1664 # Print summary. 1665 if self.quiet: 1666 print "Cheesecake index: %d (%d / %d)" % (percentage, 1667 cheesecake_index, 1668 max_cheesecake_index) 772 1669 else: 773 percent = 0 774 index_value = int(ceil(percent*100)) 775 details = "found %d/%d=%.2f%% modules/classes/methods/functions with docstrings" %\ 776 (self.docstring_cnt, self.object_cnt, percent*100) 777 self.index[index_type].value = index_value 778 self.index[index_type].details = details 779 return self.index[index_type] 780 781 def index_unittests(self): 782 """ 783 Compute unittest index as percentage of methods/functions 784 that are exercised in unit tests 785 786 Return Index object of type "unittests" 787 """ 788 unittest_cnt = 0 789 index_type = "unittests" 790 self.functions_tested = {} 791 for testfile in self.pkg_files["test"]: 792 fullpath = os.path.join(self.sandbox, testfile) 793 code = CodeParser(fullpath, self.log.debug) 794 func_called = code.functions_called() 795 self.log.debug("Functions called in unit test:") 796 self.log.debug(func_called) 797 for func in func_called: 798 self.functions_tested[func] = 1 799 self.log.debug("FUNCTIONS TO BE CHECKED WHETHER THEY ARE UNIT TESTED:") 800 self.log.debug(self.functions) 801 self.log.debug("FUNCTIONS THAT ARE UNIT TESTED:") 802 self.log.debug(self.functions_tested.keys()) 803 for funcname in self.functions: 804 if self.is_unit_tested(funcname): 805 unittest_cnt += 1 806 self.log.debug("%s is unit tested" % funcname) 807 cnt = len(self.functions) 808 if cnt: 809 percent = float(unittest_cnt)/float(cnt) 810 else: 811 percent = 0 812 index_value = int(ceil(percent*100)) 813 details = "found %d/%d=%.2f%% unit tested methods/functions" % (unittest_cnt, cnt, percent*100) 814 self.index[index_type].value = index_value 815 self.index[index_type].details = details 816 return self.index[index_type] 817 818 def is_unit_tested(self, funcname): 819 elem = funcname.split(".") 820 n1 = elem[-1] 821 n2 = "" 822 if len(elem) > 1: 823 n2 = elem[-2] + "." + elem[-1] 824 for key in self.functions_tested.keys(): 825 if key.startswith(n1) or (n2 and key.startswith(n2)): 826 return True 827 return False 828 829 def index_pylint(self): 830 """ 831 Compute pylint index as average of positive pylint scores obtained for 832 the Python files identified in the package 833 834 Return Index object of type "pylint" 835 """ 836 index_type = "pylint" 837 # Try to run the pylint script 838 rc, output = run_cmd("pylint --version") 839 if rc: 840 # We encountered an error 841 self.index[index_type].details = "pylint not properly installed" 842 return self.index[index_type] 843 index_pylint = 0 844 cnt = 0 845 for pyfile in self.pkg_files["py"]: 846 (path, filename) = os.path.split(pyfile) 847 (module, ext) = os.path.splitext(filename) 848 if module == "setup" or module == "ez_setup" or module.startswith("__"): 849 continue 850 fullpath = os.path.join(self.sandbox, pyfile) 851 self.log.debug("Running pylint on file " + fullpath) 852 rc, output = run_cmd("pylint " + fullpath) 853 if rc: 854 # We encountered an error 855 continue 856 score_line = output.split("\n")[-3] 857 s = re.search(r" (\d+\.\d+)/10", score_line) 858 # We only take positive scores into account 859 if s: 860 score = s.group(1) 861 self.log.debug("pylint score for module %s: %s" % (module, score)) 862 if score == "0.00": 863 self.log.debug("Ignoring scores of 0.00") 864 continue 865 index_pylint += float(score) 866 cnt += 1 867 if cnt: 868 avg_value = float(index_pylint)/float(cnt) 869 else: 870 avg_value = 0 871 index_value = int(ceil(avg_value*10)) 872 self.index[index_type].value = index_value 873 self.index[index_type].details = "average score is %.2f out of 10" % avg_value 874 return self.index[index_type] 875 876 def compute_cheesecake_index(self): 877 """ 878 Compute overall Cheesecake index for the package by adding up 879 specific indexes 880 """ 881 self.log.info("A given package can currently reach a MAXIMUM number of %d points" % self.max_cheesecake_index) 882 self.log.info("Starting computation of Cheesecake index for package '%s'" % (self.package)) 883 884 index_types = [] 885 #if self.name: 886 # index_types.append("pypi_download") 887 index_types.append("pypi_download") 888 if self.url: 889 index_types.append("url_download") 890 index_types += ["unpack", "unpack_dir", "install"] 891 self.cheesecake_index_installability = self.process_partial_index("INSTALLABILITY",\ 892 index_types, self.max_cheesecake_index_installability) 893 894 index_types = ["file", "dir", "docstrings"] 895 self.cheesecake_index_documentation = self.process_partial_index("DOCUMENTATION",\ 896 index_types, self.max_cheesecake_index_documentation) 897 898 index_types = [ 899 #"unittests", 900 "pylint", 901 ] 902 self.cheesecake_index_codekwalitee = self.process_partial_index("CODE KWALITEE",\ 903 index_types, self.max_cheesecake_index_codekwalitee) 904 905 print 906 self.print_separator_line("=") 907 print pad_msg("OVERALL CHEESECAKE INDEX (ABSOLUTE)", self.cheesecake_index) 908 percentage = (self.cheesecake_index * 100) / self.max_cheesecake_index 909 msg = pad_msg("OVERALL CHEESECAKE INDEX (RELATIVE)", percentage) 910 msg += " (%d out of a maximum of %d points is %d%%)" %\ 911 (self.cheesecake_index, self.max_cheesecake_index, percentage) 912 print msg 913 self.cleanup() 914 915 return self.cheesecake_index 916 917 def process_partial_index(self, partial_index_name, index_types, max_value): 918 print 919 self.log.info("Starting computation of %s index (max. points = %d)" % \ 920 (partial_index_name, max_value)) 921 partial_index_value = 0 922 for index_type in index_types: 923 partial_index_value += self.process_index(index_type) 924 925 self.print_separator_line() 926 print pad_msg("%s INDEX (ABSOLUTE)" % partial_index_name, partial_index_value) 927 percentage = (partial_index_value * 100) / max_value 928 msg = pad_msg("%s INDEX (RELATIVE)" % partial_index_name, percentage) 929 msg += " (%d out of a maximum of %d points is %d%%)" %\ 930 (partial_index_value, max_value, percentage) 931 print msg 932 return partial_index_value 933 934 def process_index(self, index_type): 935 """ 936 Compute and print index of specified type 937 """ 938 index = self.index[index_type] 939 index_method = "index_" + index_type 940 getattr(self, index_method)() 941 if not self.quiet: 942 index.print_info() 943 self.cheesecake_index += index.value 944 return index.value 945 946 def print_separator_line(self, char="-"): 947 """ 948 Print line of text, unless quiet flag was given 949 """ 950 if self.quiet: 951 return 952 print pad_line(char) 953 1670 print 1671 print pad_line("=") 1672 print pad_msg("OVERALL CHEESECAKE INDEX (ABSOLUTE)", cheesecake_index) 1673 print "%s (%d out of a maximum of %d points is %d%%)" % \ 1674 (pad_msg("OVERALL CHEESECAKE INDEX (RELATIVE)", percentage), 1675 cheesecake_index, 1676 max_cheesecake_index, 1677 percentage) 1678 1679 return cheesecake_index 1680 1681 ################################################################################ 1682 ## Command line. 1683 ################################################################################ 954 1684 955 1685 def process_cmdline_args(): 956 """ 957 Parse command-line options 1686 """Parse command-line options. 958 1687 """ 959 1688 parser = OptionParser() 1689 parser.add_option("--keep-log", action="store_true", dest="keep_log", 1690 default=False, help="don't remove log file even if run was successful") 1691 parser.add_option("--lite", action="store_true", dest="lite", 1692 default=False, help="don't run time-consuming tests (default=False)") 1693 parser.add_option("-l", "--logfile", dest="logfile", 1694 default=None, 1695 help="file to log all cheesecake messages") 960 1696 parser.add_option("-n", "--name", dest="name", 961 1697 default="", help="package name (will be retrieved via setuptools utilities, if present)") 1698 parser.add_option("-p", "--path", dest="path", 1699 default="", help="path of tar.gz/zip package on local file system") 1700 parser.add_option("-q", "--quiet", action="store_true", dest="quiet", 1701 default=False, help="only print Cheesecake index value (default=False)") 1702 parser.add_option("-s", "--sandbox", dest="sandbox", 1703 default=None, 1704 help="directory where package will be unpacked "\ 1705 "(default is to use random directory inside %s)" % tempfile.gettempdir()) 1706 parser.add_option("-t", "--static", action="store_true", dest="static", 1707 default=False, help="don't run any code from the package being tested (default=False)") 962 1708 parser.add_option("-u", "--url", dest="url", 963 1709 default="", help="package URL") 964 parser.add_option("-p", "--path", dest="path",965 default="", help="package path on local file system")966 parser.add_option("-s", "--sandbox", dest="sandbox",967 default="/tmp/cheesecake_sandbox",968 help="directory where package will be unpacked (default=/tmp/cheesecake_sandbox)")969 1710 parser.add_option("-v", "--verbose", action="store_true", dest="verbose", 970 1711 default=False, help="verbose output (default=False)") 971 parser.add_option("- q", "--quiet", action="store_true", dest="quiet",972 default=False, help=" only print Cheesecake index value (default=False)")1712 parser.add_option("-V", "--version", action="store_true", dest="version", 1713 default=False, help="Output cheesecake version and exit") 973 1714 974 1715 (options, args) = parser.parse_args() … … 976 1717 977 1718 def main(): 978 """ 979 Display Cheesecake index for package specified via command-line options 1719 """Display Cheesecake index for package specified via command-line options. 980 1720 """ 981 1721 options = process_cmdline_args() 1722 keep_log = options.keep_log 1723 lite = options.lite 1724 logfile = options.logfile 982 1725 name = options.name 1726 path = options.path 1727 quiet = options.quiet 1728 sandbox = options.sandbox 1729 static_only = options.static 983 1730 url = options.url 984 path = options.path985 sandbox = options.sandbox986 1731 verbose = options.verbose 987 quiet = options.quiet 1732 version = options.version 1733 1734 if version: 1735 print "cheesecake version %s" % VERSION 1736 sys.exit(0) 988 1737 989 1738 if not name and not url and not path: … … 992 1741 993 1742 try: 994 c = Cheesecake(name=name, url=url, path=path, sandbox=sandbox, verbose=verbose, quiet=quiet) 1743 c = Cheesecake(name=name, url=url, path=path, sandbox=sandbox, 1744 logfile=logfile, verbose=verbose, 1745 quiet=quiet, static_only=static_only, lite=lite, 1746 keep_log=keep_log) 995 1747 c.compute_cheesecake_index() 1748 c.cleanup() 996 1749 except CheesecakeError, e: 997 1750 print str(e) trunk/cheesecake/codeparser.py
r11 r150 1 import doctest 1 2 import os 3 import re 4 5 import logger 2 6 from model import System, Module, Class, Function, parseFile, processModuleAst 3 7 8 9 # Python 2.3/2.4 compatibilty hacks. 10 if getattr(doctest, 'DocTestParser', False): 11 # Python 2.4 have DocTestParser class. 12 get_doctests = doctest.DocTestParser().get_examples 13 else: 14 # Python 2.3 have _extract_examples function. 15 get_doctests = doctest._extract_examples 16 17 18 def compile_regex(pattern, user_map=None): 19 """Compile a regex pattern using default or user mapping. 20 """ 21 22 # Handy regular expressions. 23 mapping = {'ALPHA': r'[-.,?!\w]', 'WORD': r'[-.,?!\s\w]', 24 'START': r'(^|\s)', 'END': r'([.,?!\s]|$)'} 25 26 if user_map: 27 mapping = mapping.copy() 28 mapping.update(user_map) 29 30 def sub(text, mapping): 31 for From, To in mapping.iteritems(): 32 text = text.replace(From, To) 33 return text 34 35 pattern = sub(pattern, mapping) 36 37 return re.compile(pattern, re.LOCALE | re.VERBOSE) 38 39 def inline_markup(start, end=None, mapping=None): 40 if end is None: 41 end = start 42 return compile_regex(r'''(START %(start)s ALPHA %(end)s END) | 43 (START %(start)s ALPHA WORD* ALPHA %(end)s END)'''\ 44 % {'start': start, 'end': end}, mapping) 45 46 def line_markup(start, end=None): 47 return inline_markup(start, end, mapping={'ALPHA': r'[-.,?!\s\w]', 48 'START': r'(\n|^)[\ \t]*', 49 'END': r''}) 50 51 supported_formats = { 52 # reST refrence: http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html 53 'reST': [ 54 inline_markup(r'\*'), # emphasis 55 inline_markup(r'\*\*'), # strong 56 inline_markup(r'``'), # inline 57 inline_markup(r'\(', r'_\)', # hyperlink 58 {'ALPHA': r'\w', 'WORD': r'[-.\w]'}), 59 inline_markup(r'\(`', r'`_\)'), # long hyperlink 60 line_markup(r':'), # field 61 line_markup(r'[*+-]', r''), # unordered list 62 line_markup(r'((\d+) | ([a-zA-Z]+) [.\)])', r''), # ordered list 63 line_markup(r'\( ((\d+) | ([a-zA-Z]+)) \)', r''), # ordered list 64 ], 65 66 # epytext reference: http://epydoc.sourceforge.net/epytext.html 67 'epytext': [ 68 re.compile(r'[BCEGILMSUX]\{.*\}'), # inline elements 69 line_markup(r'@[a-z]+([\ \t][a-zA-Z]+)?:', r''), # fields 70 line_markup(r'-', r''), # unordered list 71 line_markup(r'\d+(\.\d+)*', r''), # ordered list 72 ], 73 74 # javadoc reference: http://java.sun.com/j2se/1.4.2/docs/tooldocs/solaris/javadoc.html 75 'javadoc': [ 76 re.compile(r'<[a-zA-z]+[^>]*>'), # HTML elements 77 line_markup(r'@[a-z][a-zA-Z]*\s', r''), # normal tags 78 re.compile(r'{@ ((docRoot) | (inheritDoc) | (link) | (linkplain) |'\ 79 ' (value)) [^}]* }', re.VERBOSE), # special tags 80 ], 81 } 82 83 84 def use_format(text, format): 85 """Return True if text includes given documentation format 86 and False otherwise. 87 88 See supported_formats for list of known formats. 89 """ 90 for pattern in supported_formats[format]: 91 if re.search(pattern, text): 92 return True 93 94 return False 95 96 4 97 class CodeParser(object): 5 """ 6 Information about the structure of a Python module 98 """Information about the structure of a Python module. 7 99 8 100 * Collects modules, classes, methods, functions and associated docstrings … … 10 102 """ 11 103 def __init__(self, pyfile, log=None): 104 """Initialize Code Parser object. 105 106 :Parameters: 107 `pyfile` : str 108 Path to a Python module to parse. 109 `log` : logger.Producer instance 110 Logger to use during code parsing. 111 """ 12 112 if log: 13 113 self.log = log.codeparser 14 114 else: 15 import logger16 115 self.log = logger.default.codeparser 17 116 self.modules = [] … … 20 119 self.method_func = [] 21 120 self.functions = [] 22 self.docstrings = {} 23 121 self.docstrings = [] # objects that have docstrings 122 self.docstrings_by_format = {} 123 self.formatted_docstrings_count = 0 124 self.doctests_count = 0 125 self.unittests_count = 0 126 127 # Initialize lists of format docstrings. 128 for format in supported_formats: 129 self.docstrings_by_format[format] = [] 130 24 131 (path, filename) = os.path.split(pyfile) 25 132 (module, ext) = os.path.splitext(filename) … … 29 136 try: 30 137 processModuleAst(parseFile(pyfile), module, self.system) 31 except: 138 except Exception, e: 139 self.log("Code parsing error occured:\n***\n%s\n***" % str(e)) 32 140 return 33 141 … … 35 143 fullname = obj.fullName() 36 144 if isinstance(obj, Module): 37 self.modules.append( obj.fullName())145 self.modules.append(fullname) 38 146 if isinstance(obj, Class): 39 self.classes.append(obj.fullName()) 147 if 'unittest.TestCase' in obj.bases or 'TestCase' in obj.bases: 148 self.unittests_count += 1 149 self.classes.append(fullname) 40 150 if isinstance(obj, Function): 41 151 self.method_func.append(fullname) 42 if obj.docstring: 43 self.docstrings[fullname] = 1 152 if isinstance(obj.docstring, str) and obj.docstring.strip(): 153 self.docstrings.append(fullname) 154 # Check docstring for known documenation formats. 155 formatted = False 156 for format in supported_formats: 157 if use_format(obj.docstring, format): 158 self.docstrings_by_format[format].append(fullname) 159 formatted = True 160 if formatted: 161 self.formatted_docstrings_count += 1 162 163 # Check if docstring include any doctests. 164 if get_doctests(obj.docstring): 165 self.doctests_count += 1 44 166 45 167 for method_or_func in self.method_func: … … 57 179 self.log("methods: " + ",".join(self.methods)) 58 180 self.log("functions: " + ",".join(self.functions)) 181 self.log("docstrings: %s" % self.docstrings_by_format) 182 self.log("number of doctests: %d" % self.doctests_count) 59 183 60 184 def object_count(self): 61 """ 62 Return number of objects found in this module 63 185 """Return number of objects found in this module. 186 187 Objects include: 64 188 * module 65 189 * classes … … 74 198 75 199 def docstring_count(self): 76 """ 77 Return number of docstrings found in this module 78 """ 79 return len(self.docstrings.keys()) 80 81 def functions_called(self): 82 """ 83 Return list of functions called by functions/methods 84 defined in this module 200 """Return number of docstrings found in this module. 201 """ 202 return len(self.docstrings) 203 204 def docstring_count_by_type(self, type): 205 """Return number of docstrings of given type found in this module. 206 """ 207 return len(self.docstrings_by_format[type]) 208 209 def _functions_called(self): 210 """Return list of functions called by functions/methods 211 defined in this module. 85 212 """ 86 213 return self.system.func_called.keys() 214 215 functions_called = property(_functions_called) trunk/cheesecake/model.py
r11 r150 1 1 """ 2 -Code borrowed from Michael Hudson's docextractor package with the author's permission. 3 -The original code is available at <http://codespeak.net/svn/user/mwh/docextractor/> 2 Code borrowed from Michael Hudson's docextractor package with the author's 3 permission. 4 5 The original code is available at http://codespeak.net/svn/user/mwh/docextractor/. 6 7 Changes: 8 * do not print warnings to stdout (in System.warning) 9 * collect all function calls 4 10 """ 11 5 12 6 13 from compiler import ast … … 11 18 import sets 12 19 13 import compiler14 20 from compiler.transformer import parse, parseFile 15 21 from compiler.visitor import walk 22 23 import ast_pp 24 25 26 def get_call_name(node): 27 assert isinstance(node, ast.CallFunc) 28 29 def get_name(node): 30 if isinstance(node, ast.CallFunc): 31 return None 32 elif isinstance(node, ast.Name): 33 return node.name 34 elif isinstance(node, str): 35 return node 36 elif isinstance(node, tuple): 37 if len(node) == 1: 38 return node[0] 39 else: 40 return "%s.%s" % (get_name(node[:-1][0]), node[-1]) 41 elif isinstance(node, ast.Getattr): 42 return get_name(node.asList()) 43 else: 44 return None 45 46 return get_name(node.node) 47 48 def get_function_calls(node, fc): 49 if not isinstance(node, ast.Node): 50 return 51 52 for child in node.getChildren(): 53 if isinstance(child, ast.CallFunc): 54 func_called = get_call_name(child) 55 if func_called: 56 fc[func_called] = 1 57 58 get_function_calls(child, fc) 59 16 60 17 61 class Documentable(object): … … 44 88 return self.parent.name2fullname(name) 45 89 90 def resolveDottedName(self, dottedname, verbose=False): 91 parts = dottedname.split('.') 92 obj = self 93 system = self.system 94 while parts[0] not in obj._name2fullname: 95 obj = obj.parent 96 if obj is None: 97 if parts[0] in system.allobjects: 98 obj = system.allobjects[parts[0]] 99 break 100 for othersys in system.moresystems: 101 if parts[0] in othersys.allobjects: 102 obj = othersys.allobjects[parts[0]] 103 break 104 else: 105 if verbose: 106 print "1 didn't find %r from %r"%(dottedname, 107 self.fullName()) 108 return None 109 break 110 else: 111 fn = obj._name2fullname[parts[0]] 112 if fn in system.allobjects: 113 obj = system.allobjects[fn] 114 else: 115 if verbose: 116 print "1.5 didn't find %r from %r"%(dottedname, 117 self.fullName()) 118 return None 119 for p in parts[1:]: 120 if p not in obj.contents: 121 if verbose: 122 print "2 didn't find %r from %r"%(dottedname, 123 self.fullName()) 124 return None 125 obj = obj.contents[p] 126 if verbose: 127 print dottedname, '->', obj.fullName(), 'in', self.fullName() 128 return obj 129 130 def dottedNameToFullName(self, dottedname): 131 if '.' not in dottedname: 132 start, rest = dottedname, '' 133 else: 134 start, rest = dottedname.split('.', 1) 135 rest = '.' + rest 136 obj = self 137 while start not in obj._name2fullname: 138 obj = obj.parent 139 if obj is None: 140 return dottedname 141 return obj._name2fullname[start] + rest 142 143 def __getstate__(self): 144 # this is so very, very evil. 145 # see doc/extreme-pickling-pain.txt for more. 146 r = {} 147 for k, v in self.__dict__.iteritems(): 148 if isinstance(v, Documentable): 149 r['$'+k] = v.fullName() 150 elif isinstance(v, list) and v: 151 for vv in v: 152 if vv is not None and not isinstance(vv, Documentable): 153 r[k] = v 154 break 155 else: 156 rr = [] 157 for vv in v: 158 if vv is None: 159 rr.append(vv) 160 else: 161 rr.append(vv.fullName()) 162 r['@'+k] = rr 163 elif isinstance(v, dict) and v: 164 for vv in v.itervalues(): 165 if not isinstance(vv, Documentable): 166 r[k] = v 167 break 168 else: 169 rr = {} 170 for kk, vv in v.iteritems(): 171 rr[kk] = vv.fullName() 172 r['!'+k] = rr 173 else: 174 r[k] = v 175 return r 46 176 47 177 class Package(Documentable): 178 kind = "Package" 48 179 def name2fullname(self, name): 49 180 raise NameError 50 181 182 51 183 class Module(Documentable): 184 kind = "Module" 52 185 def name2fullname(self, name): 53 186 if name in self._name2fullname: … … 59 192 return name 60 193 194 61 195 class Class(Documentable): 196 kind = "Class" 62 197 def setup(self): 63 198 super(Class, self).setup() 64 199 self.bases = [] 65 def __repr__(self):66 return "%s(%r, %r) # %r"%(self.__class__.__name__,67 self.name, self.shortdocstring(),68 self.bases) 200 self.rawbases = [] 201 self.baseobjects = [] 202 self.subclasses = [] 203 69 204 70 205 class Function(Documentable): 71 pass 206 kind = "Function" 207 208 209 class ModuleVistor(object): 210 def __init__(self, system, modname): 211 self.system = system 212 self.modname = modname 213 self.morenodes = [] 214 215 def default(self, node): 216 for child in node.getChildNodes(): 217 self.visit(child) 218 219 def postpone(self, docable, node): 220 self.morenodes.append((docable, node)) 221 222 def visitModule(self, node): 223 if self.system.current and self.modname in self.system.current.contents: 224 m = self.system.current.contents[self.modname] 225 assert m.docstring is None 226 m.docstring = node.doc 227 self.system.push(m, node) 228 self.default(node) 229 self.system.pop(m) 230 else: 231 if not self.system.current: 232 roots = [x for x in self.system.rootobjects if x.name == self.modname] 233 if roots: 234 mod, = roots 235 self.system.push(mod, node) 236 self.default(node) 237 self.system.pop(mod) 238 return 239 self.system.pushModule(self.modname, node.doc) 240 self.default(node) 241 self.system.popModule() 242 243 def visitClass(self, node): 244 cls = self.system.pushClass(node.name, node.doc) 245 if node.lineno is not None: 246 cls.linenumber = node.lineno 247 for n in node.bases: 248 str_base = ast_pp.pp(n) 249 cls.rawbases.append(str_base) 250 base = cls.dottedNameToFullName(str_base) 251 cls.bases.append(base) 252 self.default(node) 253 self.system.popClass() 254 255 def visitFrom(self, node): 256 modname = expandModname(self.system, node.modname) 257 name2fullname = self.system.current._name2fullname 258 for fromname, asname in node.names: 259 if fromname == '*': 260 self.system.warning("import *", modname) 261 if modname not in self.system.allobjects: 262 return 263 mod = self.system.allobjects[modname] 264 # this might fail if you have an import-* cycle, or if 265 # you're just not running the import star finder to 266 # save time (not that this is possibly without 267 # commenting stuff out yet, but...) 268 if isinstance(mod, Package): 269 self.system.warning("import * from a package", modname) 270 return 271 if mod.processed: 272 for n in mod.contents: 273 name2fullname[n] = modname + '.' + n 274 else: 275 self.system.warning("unresolvable import *", modname) 276 return 277 if asname is None: 278 asname = fromname 279 name2fullname[asname] = modname + '.' + fromname 280 281 def visitImport(self, node): 282 name2fullname = self.system.current._name2fullname 283 for fromname, asname in node.names: 284 fullname = expandModname(self.system, fromname) 285 if asname is None: 286 asname = fromname.split('.', 1)[0] 287 # aaaaargh! python sucks. 288 parts = fullname.split('.') 289 for i, part in enumerate(fullname.split('.')[::-1]): 290 if part == asname: 291 fullname = '.'.join(parts[:len(parts)-i]) 292 name2fullname[asname] = fullname 293 break 294 else: 295 name2fullname[asname] = '.'.join(parts) 296 else: 297 name2fullname[asname] = fullname 298 299 def visitFunction(self, node): 300 fc = {} 301 get_function_calls(node, fc) 302 func = self.system.pushFunction(node.name, node.doc, fc) 303 if node.lineno is not None: 304 func.linenumber = node.lineno 305 # ast.Function has a pretty lame representation of 306 # arguments. Let's convert it to a nice concise format 307 # somewhat like what inspect.getargspec returns 308 argnames = node.argnames[:] 309 kwname = starargname = None 310 if node.kwargs: 311 kwname = argnames.pop(-1) 312 if node.varargs: 313 starargname = argnames.pop(-1) 314 defaults = [] 315 for default in node.defaults: 316 try: 317 defaults.append(ast_pp.pp(default)) 318 except (KeyboardInterrupt, SystemExit): 319 raise 320 except Exception, e: 321 self.system.warning("unparseable default", "%s: %s %r"%(e.__class__.__name__, 322 e, default)) 323 defaults.append('???') 324 # argh, convert unpacked-arguments from tuples to lists, 325 # because that's what getargspec uses and the unit test 326 # compares it 327 argnames2 = [] 328 for argname in argnames: 329 if isinstance(argname, tuple): 330 argname = list(argname) 331 argnames2.append(argname) 332 func.argspec = (argnames2, starargname, kwname, tuple(defaults)) 333 self.postpone(func, node.code) 334 self.system.popFunction() 335 336 states = [ 337 'blank', 338 'preparse', 339 'importstarred', 340 'parsed', 341 'finalized', 342 ] 72 343 73 344 … … 77 348 Package = Package 78 349 Function = Function 350 ModuleVistor = ModuleVistor 79 351 80 352 def __init__(self): … … 85 357 self.rootobjects = [] 86 358 self.warnings = {} 87 # importstargraph contains edges {importe dby:[imports]} but only359 # importstargraph contains edges {importer:[imported]} but only 88 360 # for import * statements 89 361 self.importstargraph = {} 90 362 self.func_called = {} 363 self.state = 'blank' 364 self.packages = [] 365 self.moresystems = [] 366 self.urlprefix = '' 91 367 92 368 def _push(self, cls, name, docstring): … … 128 404 The default is that the second definition "wins". 129 405 ''' 406 i = 0 407 fn = obj.fullName() 408 while (fn + ' ' + str(i)) in self.allobjects: 409 i += 1 410 prev = self.allobjects[obj.fullName()] 411 prev.name = obj.name + ' ' + str(i) 412 self.allobjects[prev.fullName()] = prev 130 413 self.warning("duplicate", self.allobjects[obj.fullName()]) 131 414 self.allobjects[obj.fullName()] = obj … … 204 487 205 488 def warning(self, type, detail): 206 fn = self.current.fullName() 207 #print fn, type, detail 489 if self.current is not None: 490 fn = self.current.fullName() 491 else: 492 fn = '<None>' 208 493 self.warnings.setdefault(type, []).append((fn, detail)) 209 494 … … 212 497 if isinstance(o, cls): 213 498 yield o 499 500 def finalStateComputations(self): 501 self.recordBasesAndSubclasses() 502 503 def recordBasesAndSubclasses(self): 504 for cls in self.objectsOfType(Class): 505 for n in cls.bases: 506 o = cls.parent.resolveDottedName(n) 507 cls.baseobjects.append(o) 508 if o: 509 o.subclasses.append(cls) 510 511 def __getstate__(self): 512 state = self.__dict__.copy() 513 del state['moresystems'] 514 return state 515 516 def __setstate__(self, state): 517 self.moresystems = [] 518 # this is so very, very evil. 519 # see doc/extreme-pickling-pain.txt for more. 520 self.__dict__.update(state) 521 for obj in self.orderedallobjects: 522 for k, v in obj.__dict__.copy().iteritems(): 523 if k.startswith('$'): 524 del obj.__dict__[k] 525 obj.__dict__[k[1:]] = self.allobjects[v] 526 elif k.startswith('@'): 527 n = [] 528 for vv in v: 529 if vv is None: 530 n.append(None) 531 else: 532 n.append(self.allobjects[vv]) 533 del obj.__dict__[k] 534 obj.__dict__[k[1:]] = n 535 elif k.startswith('!'): 536 n = {} 537 for kk, vv in v.iteritems(): 538 n[kk] = self.allobjects[vv] 539 del obj.__dict__[k] 540 obj.__dict__[k[1:]] = n 541 214 542 215 543 def expandModname(system, modname, givewarning=True): … … 242 570 modname = expandModname(self.system, node.modname, False) 243 571 self.system.importstargraph.setdefault( 244 modname, []).append(self.modfullname) 245 246 class ModuleVistor(object): 247 def __init__(self, system, modname): 248 self.system = system 249 self.modname = modname 250 self.morenodes = [] 251 252 def default(self, node): 253 for child in node.getChildNodes(): 254 self.visit(child) 255 256 def postpone(self, docable, node): 257 self.morenodes.append((docable, node)) 258 259 def visitModule(self, node): 260 if self.system.current and self.modname in self.system.current.contents: 261 m = self.system.current.contents[self.modname] 262 assert m.docstring is None 263 m.docstring = node.doc 264 self.system.push(m, node) 265 self.default(node) 266 self.system.pop(m) 267 else: 268 self.system.pushModule(self.modname, node.doc) 269 self.default(node) 270 self.system.popModule() 271 272 def visitClass(self, node): 273 cls = self.system.pushClass(node.name, node.doc) 274 for n in node.bases: 275 if isinstance(n, ast.Name): 276 cls.bases.append(cls.parent.name2fullname(n.name)) 277 elif isinstance(n, ast.Getattr): 278 p = [] 279 while isinstance(n, ast.Getattr): 280 p.append(n.attrname) 281 n = n.expr 282 assert isinstance(n, ast.Name) 283 p.append(cls.parent.name2fullname(n.name)) 284 p.reverse() 285 assert None not in p, n 286 cls.bases.append('.'.join(p)) 287 else: 288 assert not n 289 self.default(node) 290 self.system.popClass() 291 292 def visitFrom(self, node): 293 modname = expandModname(self.system, node.modname) 294 name2fullname = self.system.current._name2fullname 295 for fromname, asname in node.names: 296 if fromname == '*': 297 self.system.warning("import *", modname) 298 if modname not in self.system.allobjects: 299 return 300 mod = self.system.allobjects[modname] 301 #snarl (see below) 302 #assert mod.processed 303 self.system.warning("mwh is an idiot", "") 304 for n in mod.contents: 305 name2fullname[n] = modname + '.' + n 306 return 307 if asname is None: 308 asname = fromname 309 name2fullname[asname] = modname + '.' + fromname 310 311 def visitImport(self, node): 312 name2fullname = self.system.current._name2fullname 313 for fromname, asname in node.names: 314 fullname = expandModname(self.system, fromname) 315 if asname is None: 316 asname = fromname.split('.', 1)[0] 317 # aaaaargh! python sucks. 318 parts = fullname.split('.') 319 for i, part in enumerate(fullname.split('.')[::-1]): 320 if part == asname: 321 fullname = '.'.join(parts[:len(parts)-i]) 322 name2fullname[asname] = fullname 323 break 324 else: 325 name2fullname[asname] = '.'.join(parts) 326 else: 327 name2fullname[asname] = fullname 328 329 330 def visitFunction(self, node): 331 fc = {} 332 get_function_calls(node, fc) 333 #print fc.keys() 334 func = self.system.pushFunction(node.name, node.doc, fc) 335 # ast.Function has a pretty lame representation of 336 # arguments. Let's convert it to a nice concise 337 # getargspec-like format and include it in the Function 338 # object. 339 argnames = node.argnames[:] 340 kwname = starargname = None 341 if node.kwargs: 342 kwname = argnames.pop(-1) 343 if node.varargs: 344 starargname = argnames.pop(-1) 345 defaults = [] 346 for default in node.defaults: 347 if isinstance(default, ast.Const): 348 defaults.append(default.value) 349 elif isinstance(default, ast.Name): 350 defaults.append(default.name) 351 else: 352 self.system.warning("unparseable default", repr(default)) 353 defaults.append('???') 354 #assert False, "don't know how to handle default %r"%(default,) 355 # argh, convert unpacked-arguments from tuples to lists, 356 # because that's what getargspec uses and the unit test 357 # compares it 358 argnames2 = [] 359 for argname in argnames: 360 if isinstance(argname, tuple): 361 argname = list(argname) 362 argnames2.append(argname) 363 func.argspec = (argnames2, starargname, kwname, tuple(defaults)) 364 #for child in node.getChildren(): 365 # if isinstance(child, compiler.ast.Stmt): 366 # for c in child.getChildren(): 367 # print c.__class__ 368 # print c 369 self.postpone(func, node.code) 370 self.system.popFunction() 371 372 def get_function_calls(node, fc): 373 if not isinstance(node, compiler.ast.Node): 374 return 375 for child in node.getChildren(): 376 #print "child:", child 377 if isinstance(child, compiler.ast.CallFunc): 378 funcname = "" 379 attrname = "" 380 n = child.node 381 #print "n:", n 382 #print n.__class__ 383 if isinstance(n, compiler.ast.Getattr): 384 expr = n.expr 385 if isinstance(expr, compiler.ast.Name): 386 funcname = expr.name 387 attrname = n.attrname 388 func_called = "" 389 if funcname: func_called = funcname + "." 390 func_called += attrname 391 if func_called: 392 fc[func_called] = 1 393 get_function_calls(child, fc) 394 572 self.modfullname, []).append(modname) 573 395 574 def processModuleAst(ast, name, system): 396 mv = ModuleVistor(system, name)575 mv = system.ModuleVistor(system, name) 397 576 walk(ast, mv) 398 577 while mv.morenodes: … … 405 584 def fromText(src, modname='<test>', system=None): 406 585 if system is None: 407 system = System() 408 processModuleAst(parse(src), modname, system) 409 return system.rootobjects[0] 586 _system = System() 587 else: 588 _system = system 589 processModuleAst(parse(src), modname, _system) 590 if system is None: 591 _system.finalStateComputations() 592 return _system.rootobjects[0] 410 593 411 594 412 595 def preprocessDirectory(system, dirpath): 413 package = system.pushPackage(os.path.basename(dirpath), None) 596 assert system.state in ['blank', 'preparse'] 597 if os.path.basename(dirpath): 598 package = system.pushPackage(os.path.basename(dirpath), None) 599 else: 600 package = None 414 601 for fname in os.listdir(dirpath): 415 602 fullname = os.path.join(dirpath, fname) 416 if os.path.isdir(fullname) and os.path.exists(os.path.join(fullname, '__init__.py')) :603 if os.path.isdir(fullname) and os.path.exists(os.path.join(fullname, '__init__.py')) and fname != 'test': 417 604 preprocessDirectory(system, fullname) 418 605 elif fname.endswith('.py'): … … 422 609 mod.processed = False 423 610 system.popModule() 424 system.popPackage() 425 426 def processDirectory(system, dirpath): 427 preprocessDirectory(system, dirpath) 611 if package: 612 system.popPackage() 613 system.state = 'preparse' 614 615 def findImportStars(system): 616 assert system.state in ['preparse'] 428 617 modlist = list(system.objectsOfType(Module)) 429 618 for mod in modlist: 430 619 system.push(mod.parent) 431 620 isf = ImportStarFinder(system, mod.fullName()) 432 walk(parseFile(mod.filepath), isf) 621 try: 622 ast = parseFile(mod.filepath) 623 except (SyntaxError, ValueError): 624 system.warning("cannot parse", mod.filepath) 625 walk(ast, isf) 433 626 system.pop(mod.parent) 434 435 # snarl; a toposort is meant to go here. 436 newlist = modlist 627 system.state = 'importstarred' 628 629 def extractDocstrings(system): 630 assert system.state in ['preparse', 'importstarred'] 631 # and so much more... 632 modlist = list(system.objectsOfType(Module)) 633 newlist = toposort([m.fullName() for m in modlist], system.importstargraph) 437 634 438 635 for mod in newlist: 636 mod = system.allobjects[mod] 439 637 system.push(mod.parent) 440 processModuleAst(parseFile(mod.filepath), mod.name, system) 638 try: 639 ast = parseFile(mod.filepath) 640 except (SyntaxError, ValueError): 641 system.warning("cannot parse", mod.filepath) 642 processModuleAst(ast, mod.name, system) 441 643 mod.processed = True 442 644 system.pop(mod.parent) 443 444 def main(argv): 645 system.state = 'parsed' 646 647 def finalStateComputations(system): 648 assert system.state in ['parsed'] 649 system.finalStateComputations() 650 system.state = 'finalized' 651 652 def processDirectory(system, dirpath): 653 preprocessDirectory(system, dirpath) 654 findImportStars(system) 655 extractDocstrings(system) 656 finalStateComputations(system) 657 658 def toposort(input, edges): 659 # this doesn't detect cycles in any clever way. 660 output = [] 661 input = dict.fromkeys(input) 662 def p(i): 663 for j in edges.get(i, []): 664 if j in input: 665 del input[j] 666 p(j) 667 output.append(i) 668 while input: 669 p(input.popitem()[0]) 670 return output 671 672 673 def main(systemcls, argv): 445 674 if '-r' in argv: 446 675 argv.remove('-r') 447 676 assert len(argv) == 1 448 system = System()677 system = systemcls() 449 678 processDirectory(system, argv[0]) 450 679 pickle.dump(system, open('da.out', 'wb'), pickle.HIGHEST_PROTOCOL) … … 454 683 print k, len(v) 455 684 else: 456 system = System()685 system = systemcls() 457 686 for fname in argv: 458 687 modname = os.path.splitext(os.path.basename(fname))[0] # XXX! … … 463 692 464 693 if __name__ == '__main__': 465 main( sys.argv[1:])694 main(System, sys.argv[1:]) trunk/cheesecake/subprocess.py
r10 r150 394 394 import pickle 395 395 396 __all__ = ["Popen", "PIPE", "STDOUT", "call" ]396 __all__ = ["Popen", "PIPE", "STDOUT", "call", "ProcessError"] 397 397 398 398 try: … … 495 495 return ''.join(result) 496 496 497 class ProcessError(Exception): 498 """This exception is raised when there is an error calling 499 a subprocess.""" 500 pass 497 501 498 502 class Popen(object): … … 984 988 if data != "": 985 989 child_exception = pickle.loads(data) 986 raise child_exception990 raise ProcessError, child_exception 987 991 988 992 trunk/setup.py
r5 r150 1 1 #! /usr/bin/env python 2 2 import sys 3 import os .path3 import os 4 4 5 5 from setuptools import setup 6 6 from pkg_resources import require 7 from cheesecake import __version__ as VERSION 8 9 # Instruct nose to use doctests. 10 os.environ['NOSE_WITH_DOCTEST'] = 'True' 11 os.environ['NOSE_DOCTEST_TESTS'] = 'True' 12 os.environ['NOSE_INCLUDE'] = 'unit' 13 os.environ['NOSE_INCLUDE_EXE'] = 'True' 7 14 8 15 setup( 9 16 name = 'Cheesecake', 10 version = '0.1',17 version = VERSION, 11 18 12 19 # metadata for upload to PyPI 13 author = "Grig Gheorghiu ",14 author_email = "grig@gheorghiu.net ",20 author = "Grig Gheorghiu and Michal Kwiatkowski", 21 author_email = "grig@gheorghiu.net and ruby@joker.linuxstuff.pl", 15 22 description = 'Computes "goodness" index for Python packages based on various empirical "kwalitee" factors', 16 23 license = "PSF", 17 24 keywords = "cheesecake quality index kwalitee cheeseshop pypi", 18 url = "http:// tracos.org/cheesecake",25 url = "http://pycheesecake.org/", 19 26 20 27 packages = ['cheesecake', … … 22 29 scripts = ['cheesecake_index', 23 30 ], 31 entry_points = { 32 'console_scripts': [ 33 'cheesecake_index = cheesecake.cheesecake_index:main', 34 ] 35 }, 24 36 test_suite = 'nose.collector', 25 )37 ) trunk/tests/data/module1.py
r8 r150 1 1 """ 2 2 Docstring for module1 3 4 @summary: Code used inside test_code_parser.py unit test. 3 5 """ 4 6 … … 6 8 """ 7 9 Docstring for Class1 10 11 @see how.Tests#are(performed) 8 12 """ 9 13 10 14 def __init__(self): 11 15 """ 12 Methods starting with __ are not kipped16 Methods starting with __ are not skipped 13 17 """ 14 18 pass … … 36 40 pass 37 41 42 def method5(self): 43 """Method with few definitions. 44 45 :Word: And its definition. 46 """ 47 pass 48 38 49 class Class2: 39 50 … … 42 53 """ 43 54 pass 55 44 56 45 57 def func1(): … … 66 78 """ 67 79 return 80 81 def func6(): 82 """ 83 """ 84 pass 85 86 def func7(): 87 "Time to get *a bit* of reST." 88 pass 89 90 def func8(argument): 91 """This is test function for the epytext parser. 92 93 @param argument: And you really can't say if this is 94 epytext or javadoc! We count both. 95 """ 96 pass 97 98 99 class Class3(object): 100 """ 101 New-style class with epytext link: U{http://pycheesecake.org}. 102 """ 103 pass 104 105 106 def outer_function(*args): 107 x = 42 108 109 def inner_function(): 110 """Short docstring.""" 111 pass 112 113 return x
