Description
The increasing frequency of attacks on the software supply chain has highlighted the
importance of software bills of materials (SBOMs), especially in widely used ecosystems
such as PHP. However, there are few studies that evaluate the accuracy and
completeness of SBOM generation tools specifically for PHP projects. This study fills
this gap with a comprehensive qualitative and quantitative analysis of four well-known
SBOM generation tools: Syft, Trivy, CycloneDX PHP Composer Plugin, and GitHub
Dependency Graph.
Our qualitative analysis revealed significant discrepancies, including inconsistent handling
of dependencies, limited support for resolving missing versions, issues with
naming conventions, and limited metadata support. The quantitative evaluation also
showed significant differences in the number of dependencies identified, the frequency
of duplicate entries, and overall accuracy, as measured by precision, recall, and F1
scores using a Ground Truth constructed from the composer.lock file. Among the
tools evaluated, the CycloneDX PHP Composer plugin showed the highest accuracy,
achieving an F1 score of approximately 0.93 due to its effective use of metadata and
explicitly eliminating duplicates. However, it can only be used in PHP projects that
have Composer integrated. Syft can also be used without Composer and found a large
number of dependencies, but also a high number of duplicates.
The results highlight critical areas where existing SBOM tools need improvement and
underscore the need for standardised approaches to dependency identification and
version resolution. This research provides fundamental insights and recommendations
for improving the accuracy and usability of SBOMs, contributing to strengthening the
software supply chain security within the PHP ecosystem.
|
