gnemoug/distribute_crawler
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
- Repository: https://github.com/gnemoug/distribute_crawler
- Commit:
fb74ad853126b3a6358f987d763d492d8f3eec12 - Generated: 2026-11-13T23:22:51+00:00
- Scorer 0.1.0, scoring prov-4+qual-2+sec-2, report schema v2
Scores
| Pillar | Score | Signal | Confidence | Coverage |
|---|---|---|---|---|
| AI provenance (estimated AI-generated share) | 0 | none-detected | 44% | 100% |
| Code quality | 46 | poor | 90% | 100% |
| Security | 53 | weak | 85% | 100% |
AI provenance
No AI attribution or AI-consistent velocity signals detected.
- Explicitly attributed share (lower bound): 0.0%
- Unattributed share in AI-consistent velocity bursts: 0.0%
- Estimated AI-generated share: 0.0%
| Month | Lines added | Attributed to AI |
|---|---|---|
| 2013-04 | 2,636 | 0% |
| 2013-08 | 1 | 0% |
AI provenance describes how code was produced; it is not a quality judgment. Unless attribution is explicit, these are signals consistent with AI generation, not proof. Squash/rebase workflows, imported code, and rewritten history reduce accuracy.
Code quality
Code quality is poor: test-to-source ratio 0.01, no CI test run detected; 0% of function code in complex or long functions; 19% duplicated lines. Weakest area: lint.
| Area | Score |
|---|---|
| Complexity (lizard) | 100 |
| Duplication | 31 |
| Code smells (Opengrep) | 18 |
| Tests and CI | 31 |
| Documentation | 40 |
| Type safety | 40 |
| Process (Scorecard review/maintenance) | n/a |
- Functions: 150; mean complexity 2.2, 90th percentile 5; 0 complex and 0 long functions
- Duplicated lines: 19.5% of 1,850 compared
- Tests: 30 test lines for 2,604 source lines (ratio 0.01); CI test commands: none detected; 25% of recent source commits also change tests
- Docs missing or thin: license, contributing, changelog
Security
Security posture is weak: 8 high-severity SAST findings. Weakest area: posture.
| Area | Score |
|---|---|
| Secrets (gitleaks, tree + history) | 100 |
| Static analysis (Opengrep) | 22 |
| Vulnerable dependencies (OSV) | n/a |
| CI workflows (zizmor) | n/a |
| Posture (policy, updates, signing, lockfiles) | 0 |
| OpenSSF Scorecard | n/a |
- Posture checks passed: lockfiles; missing: security policy, dependency updates, signed commits
Top findings
| Severity | Finding | Location |
|---|---|---|
| high | opengrep/python.lang.security.audit.dangerous-system-call-audit: Found dynamic content used in a system call. This is dangerous if external data can reach this function call because it allows a malicious actor to execute commands. Use the 'subprocess' module instead, which is easier to use without accidentally exposing a command injection vulnerability. [confidence low] |
woaidu_crawler/woaidu_crawler/commands/compression.py:21 |
| high | opengrep/python.lang.security.audit.dangerous-system-call-audit: Found dynamic content used in a system call. This is dangerous if external data can reach this function call because it allows a malicious actor to execute commands. Use the 'subprocess' module instead, which is easier to use without accidentally exposing a command injection vulnerability. [confidence low] |
woaidu_crawler/woaidu_crawler/commands/decomperssion.py:33 |
| high | opengrep/python.lang.security.audit.dangerous-subprocess-use-audit: Detected subprocess function 'Popen' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'. [confidence low] |
woaidu_crawler/woaidu_crawler/commands/init_sharding_mongodb.py:272 |
| high | opengrep/python.lang.security.audit.dangerous-subprocess-use-tainted-env-args: Detected subprocess function 'Popen' with user controlled data. A malicious actor could leverage this to perform command injection. You may consider using 'shlex.quote()'. [confidence medium] |
woaidu_crawler/woaidu_crawler/commands/init_sharding_mongodb.py:272 |
| high | opengrep/python.lang.security.audit.dangerous-subprocess-use-audit: Detected subprocess function 'Popen' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'. [confidence low] |
woaidu_crawler/woaidu_crawler/commands/init_sharding_mongodb.py:284 |
| high | opengrep/python.lang.security.audit.dangerous-subprocess-use-tainted-env-args: Detected subprocess function 'Popen' with user controlled data. A malicious actor could leverage this to perform command injection. You may consider using 'shlex.quote()'. [confidence medium] |
woaidu_crawler/woaidu_crawler/commands/init_sharding_mongodb.py:284 |
| high | opengrep/python.lang.security.audit.dangerous-subprocess-use-audit: Detected subprocess function 'Popen' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'. [confidence low] |
woaidu_crawler/woaidu_crawler/commands/init_sharding_mongodb.py:303 |
| high | opengrep/python.lang.security.audit.dangerous-subprocess-use-tainted-env-args: Detected subprocess function 'Popen' with user controlled data. A malicious actor could leverage this to perform command injection. You may consider using 'shlex.quote()'. [confidence medium] |
woaidu_crawler/woaidu_crawler/commands/init_sharding_mongodb.py:303 |
| medium | opengrep/python.lang.security.insecure-hash-algorithm-sha1: Detected SHA1 hash algorithm which is considered insecure. SHA1 is not collision resistant and is therefore not suitable as a cryptographic signature. Use SHA256 or SHA3 instead. [confidence medium] |
woaidu_crawler/woaidu_crawler/pipelines/file.py:242 |
| medium | opengrep/python.lang.security.insecure-hash-algorithm-sha1: Detected SHA1 hash algorithm which is considered insecure. SHA1 is not collision resistant and is therefore not suitable as a cryptographic signature. Use SHA256 or SHA3 instead. [confidence medium] |
woaidu_crawler/woaidu_crawler/pipelines/mongodb_book_file.py:257 |
| medium | opengrep/python.lang.security.deserialization.avoid-cPickle: Avoid using cPickle, which is known to lead to code execution vulnerabilities. When unpickling, the serialized data could be manipulated to run arbitrary code. Instead, consider serializing the relevant data as JSON or a similar text-based serialization format. [confidence low] |
woaidu_crawler/woaidu_crawler/scrapy_redis/queue.py:29 |
| medium | opengrep/python.lang.security.deserialization.avoid-pickle: Avoid using pickle, which is known to lead to code execution vulnerabilities. When unpickling, the serialized data could be manipulated to run arbitrary code. Instead, consider serializing the relevant data as JSON or a similar text-based serialization format. [confidence low] |
woaidu_crawler/woaidu_crawler/scrapy_redis/queue.py:29 |
| medium | opengrep/python.lang.security.deserialization.avoid-cPickle: Avoid using cPickle, which is known to lead to code execution vulnerabilities. When unpickling, the serialized data could be manipulated to run arbitrary code. Instead, consider serializing the relevant data as JSON or a similar text-based serialization format. [confidence low] |
woaidu_crawler/woaidu_crawler/scrapy_redis/queue.py:33 |
| medium | opengrep/python.lang.security.deserialization.avoid-pickle: Avoid using pickle, which is known to lead to code execution vulnerabilities. When unpickling, the serialized data could be manipulated to run arbitrary code. Instead, consider serializing the relevant data as JSON or a similar text-based serialization format. [confidence low] |
woaidu_crawler/woaidu_crawler/scrapy_redis/queue.py:33 |
Static signals only: the scorer never runs the project's code. Secret findings are pattern matches that were not verified, and secret values are never stored.
Repository profile
- Source lines: 2,604; test lines: 30
| Language | Lines |
|---|---|
| Python | 2,634 |