c9eef2d697
Most tools used for compliance and SBOM generation use SPDX identifiers This change brings us a step closer to an easy SBOM generation. Signed-off-by: Alin Jerpelea <alin.jerpelea@sony.com> |
||
|---|---|---|
| .. | ||
| apachize.py | ||
| author_mappings.json | ||
| check.py | ||
| log2json.sh | ||
| README.md | ||
Licensing Check Scripts
This directory holds various scripts to aid in clearing IP on files. The main script is
log2json which should receive a path to a file (relative to repository root). It will retrieve
the file history and build a JSON output with all metadata needed for analysis. The second
script is check.py which receives a JSON file generated from the previous tool (either from a file
or from stdin, using - for the filename).
The check script will:
- retrieve git commit authors
- parse commit message for possible attributions ("authored by: ...", among other variations)
- retrieve file contents at each commit, parse the license header and try to extract authors and companies (copyrights) listed there
Steps 2 and 3 are based on heuristics. The attributions may not match the regular expressions used so there may be misdetections. Authors on headers are easier to detect. In fact, this will pick up various false positives (non-author strings) which will have to be ignored by the user.
All of these authorship information is aggregated and in a final step, the names are used
to check for ICLAs, based on the ICLA databases (see below), which need to be manually downloaded.
If a given author name is not matched, their email searched for in the author_mappings.json file,
which is a dictionary of email to real name. This allows to handle users with alternative email
addresses.
The script output will report a green check if author matched the ICLA database or a red cross if not. Note that given the false positives in steps 2 and 3, there may be both non-author strings that obviously do not match and also there may be an attribution which was not detected in a commit message. The thorough approach would be to run the check script with verbosity ('-v') which will print the metadata of each commit, including the commit message. If double verbosity is used ('-vv'), the whole file will be printed, which allows to check the header.
Inaccessible blobs
Since some files in the repositories lived during some part of their history in a separate repository (linked as a submodule to main repo), their blobs (basically the file at a given point in time) will not be accessible. This means that the file at that point in time cannot be accessed for analyzing its header.
Zero blob hash
Some blob hashes will be all zeros, which means that the file was deleted at this point in time. Sometimes this is due to merges or renames (which may be part of the moving in and out of submodules).
ICLA database
In order to retrieve a list of all users with CLAs, download the following files:
- https://whimsy.apache.org/public/icla-info.json
- https://whimsy.apache.org/public/icla-info_noid.json
There are two files since not all users with CLAs have Apache IDs. These lists do not contain emails, but a manual search form is also here: