One PDF Per Domain in a Bulk Scan — and How to Re-Group on the Fly

Introduction to Domain-Based PDF Reporting

As security professionals, we understand the intricacies involved in performing bulk scans, especially when it comes to reporting. Traditionally, generating a single comprehensive report for all identified vulnerabilities across multiple domains often leads to cumbersome and less actionable insights. This method can obscure domain-specific issues, making it challenging for teams to prioritize and address vulnerabilities efficiently. The need for domain-specific reporting has become increasingly apparent, allowing teams to tailor their remediation strategies for each domain.

Traditional bulk reporting methods face significant challenges, such as the overwhelming volume of data and the difficulty in segregating domain-specific vulnerabilities. This often results in reports that are either too lengthy or lack the specificity needed for effective mitigation. Furthermore, the manual effort required to sift through data and create domain-specific reports is labor-intensive and prone to human error. These challenges highlight the necessity for a more structured and automated approach to reporting.

def generate_pdf_report(domain_data):
    from reportlab.lib.pagesizes import letter
    from reportlab.pdfgen import canvas

    for domain, data in domain_data.items():
        file_name = f"{domain}_report.pdf"
        c = canvas.Canvas(file_name, pagesize=letter)
        c.drawString(100, 750, f"Report for {domain}")
        for vuln in data['vulnerabilities']:
            c.drawString(100, 730, f"- {vuln['id']}: {vuln['description']}")
            c.showPage()
        c.save()
    return "PDF reports generated."

To address these challenges, Pentestas has introduced a solution that generates one PDF report per domain, streamlining the process and ensuring that each report is concise and focused. Our system automatically segregates vulnerabilities by domain, producing tailored reports that highlight domain-specific risks and recommended actions. This approach not only enhances the clarity of the reports but also significantly reduces the time and effort required for teams to digest the information and act upon it. By enabling on-the-fly re-grouping, our platform offers flexibility and adaptability in how reports are generated and utilized.

Technical Architecture of PDF Generation

Our PDF generation process stands on a robust backend architecture designed to handle bulk scan results efficiently. The system begins with a distributed queue that assigns scan results to specific workers based on domain. Each worker is responsible for fetching scan data, processing it, and ultimately generating a PDF report. This approach ensures that each domain's data is isolated and processed independently, reducing the risk of data leakage. By decoupling data retrieval and processing, we enhance the reliability and maintainability of the system.

We utilize a microservices architecture to achieve scalable and efficient PDF processing. Each microservice is a specialized component responsible for a specific task, such as data fetching, PDF creation, or storage. This modular design allows us to scale individual components based on demand without affecting others. For instance, during peak load times, we can dynamically scale the PDF generation service to handle increased throughput. This flexibility is crucial for maintaining performance while processing potentially thousands of PDFs simultaneously.

Integration with the existing Pentestas infrastructure was seamless, thanks to our use of standard APIs and shared authentication mechanisms. We leveraged existing logging and monitoring frameworks to ensure that every part of the PDF generation pipeline is observable and accountable. This integration allows us to maintain a unified view of all operations, with logs accessible via our central dashboard. Additionally, by reusing existing authentication systems, we ensure that our microservices communicate securely and efficiently, keeping all data transactions safe.

Implementing Apex-Domain Grouping

Defining the apex-domain is essential for structuring our reports in a meaningful way. An apex-domain represents the highest level of a domain's hierarchy, excluding subdomains and protocol prefixes. By focusing on the apex-domain, we ensure that all related subdomains are grouped together in a single PDF report. This approach allows us to provide a comprehensive overview of vulnerabilities and configurations for each domain owner, making the findings more actionable.

Our algorithm for grouping subdomains begins by identifying the apex-domain for each URL. We utilize a combination of regular expressions and the tldextract library to accurately parse domain structures. Once identified, subdomains are mapped to their corresponding apex-domains in a dictionary. This mapping allows us to efficiently group and organize our scan results.

from tldextract import extract

def get_apex_domain(url):
    result = extract(url)
    return f"{result.domain}.{result.suffix}"

urls = ["sub.example.com", "example.com", "test.example.org"]
domain_mapping = {}

for url in urls:
    apex = get_apex_domain(url)
    if apex not in domain_mapping:
        domain_mapping[apex] = []
    domain_mapping[apex].append(url)

print(domain_mapping)

Handling exceptions and edge cases is a critical part of our domain grouping strategy. Domains with multiple suffixes, like co.uk or ac.jp, require special handling to ensure they are not misclassified. Additionally, we account for wildcard domains and malformed URLs through robust error handling mechanisms. This ensures that our grouping remains accurate and reliable, even when faced with complex or unexpected input.

Per-Subdomain Mode: A Deep Dive

The per-subdomain mode in our bulk scanning workflow offers a granular approach by generating individual reports for each subdomain. This is particularly useful in environments where subdomains serve distinct purposes, as it allows for tailored insights into each segment. For instance, in a corporate setup where api.example.com and mail.example.com have different security postures, this mode helps isolate vulnerabilities specific to each. This isolation is crucial for teams that need to prioritize fixes based on the subdomain's operational criticality.

One of the key impacts of per-subdomain mode is the enhancement of report granularity. By focusing on individual subdomains, the insights gained are far more detailed than those from a monolithic domain report. We often find that this approach uncovers specific vulnerabilities that might be overlooked in a broader report. For example, a Cross-Site Scripting (CVE-2023-0456) vulnerability might appear only on a staging subdomain, allowing developers to address it without disrupting the live environment. However, the challenge lies in efficiently balancing detail with readability.

for subdomain in subdomains_list:
    vulnerabilities = scan_subdomain(subdomain)
    report = generate_report(subdomain, vulnerabilities)
    save_report(f"/reports/{subdomain}.pdf", report)

Balancing the level of detail in subdomain reports with readability is akin to walking a tightrope. Too much information can overwhelm, while too little might miss critical vulnerabilities. Our solution is to include executive summaries that highlight key findings and recommended actions, followed by detailed sections for technical teams. This structure ensures that C-suite executives and security analysts can both derive value from the same report. Additionally, feedback loops from our users have been instrumental in refining report formats to maximize clarity and impact.

Full-Tenant Fernet Decrypt: Ensuring Security

Fernet encryption is a crucial component of Pentestas, providing a mechanism to securely encode and decode sensitive information. In our platform, we utilize Fernet to encrypt data at rest, ensuring that only authorized users can access it. This symmetric encryption method uses the same key for both encryption and decryption, making key management a critical aspect. The Fernet module in Python’s cryptography package is our go-to tool, as it guarantees that a message encrypted and then decrypted results in the original message.

When generating reports, particularly in a multi-tenant environment, full-tenant decryption becomes necessary. Each tenant’s data must be processed to produce comprehensive reports without compromising individual security. This involves decrypting data for analysis and re-encrypting it once the report is generated. Our backend processes handle this efficiently, ensuring that tenants receive accurate and timely reports. The decrypted data is never stored in its raw form; it is held temporarily in volatile memory to prevent leaks.

Implementing security measures for safe decryption is a top priority. We enforce strict access controls and audit logging to monitor decryption events. Additionally, our infrastructure is designed to isolate decryption operations in secure environments, minimizing the attack surface. The following code snippet demonstrates how we utilize Fernet for decrypting tenant data:

from cryptography.fernet import Fernet

# Load the encryption key
with open('/etc/pentestas/keys/tenant.key', 'rb') as key_file:
    key = key_file.read()

fernet = Fernet(key)

# Decrypt the data
encrypted_data = b'gAAAAABh...'  # Encrypted data here
plaintext = fernet.decrypt(encrypted_data)
print("Decrypted data:", plaintext.decode())

This code highlights the steps taken to decrypt data securely. We ensure that the encryption keys are stored securely and accessed only by authorized services. By maintaining these rigorous security protocols, Pentestas can deliver reliable reports while safeguarding tenant data integrity and confidentiality.

Dynamic Regrouping on the Fly

In the realm of cybersecurity assessments, the need for flexibility in report generation is paramount. Once our system generates individual PDFs for each domain during a bulk scan, the challenge lies in enabling dynamic regrouping. We employ a mechanism that allows us to reassemble these reports post-generation, based on new criteria or insights that arise. Through the use of metadata tags embedded within each PDF, we can swiftly reorganize content without having to regenerate the entire report set. This not only saves processing time but also provides our users with the agility needed to adapt to evolving threat landscapes.

User interface plays a critical role in facilitating dynamic regrouping. Our design ethos prioritizes intuitive interaction, ensuring that even complex regrouping tasks can be executed seamlessly. The interface offers drag-and-drop functionality, allowing users to visually cluster related reports. Additionally, we provide filtering options to refine selections based on specific criteria such as vulnerability severity or domain risk level. This user-centric approach minimizes friction and enhances the overall experience, making it easier to manage large-scale assessments efficiently.

def regroup_reports(pdf_list, criteria):
    grouped_reports = {}
    for pdf in pdf_list:
        key = extract_metadata(pdf, criteria)
        if key not in grouped_reports:
            grouped_reports[key] = []
        grouped_reports[key].append(pdf)
    return grouped_reports

Implementing real-time adjustments poses its own set of technical challenges. Our architecture is designed to handle these on-the-fly changes without disrupting service performance. We utilize asynchronous processing to ensure that regrouping operations do not interfere with ongoing scans. This is achieved by leveraging technologies such as Redis for in-memory data storage and Kafka for streaming updates. These technologies allow for scalable and responsive adjustments, ensuring that Pentestas can maintain its high standards of reliability and speed.

Case Studies and Real-World Applications

Our journey with clients implementing the one-PDF-per-domain strategy has been immensely rewarding. One notable example involves a fintech company managing over 500 domains. By integrating our automated PDF generation into their existing workflows, they observed a 30% reduction in time spent on weekly security reviews. Previously, their team had to manually compile findings from multiple sources, but now, each domain's vulnerabilities are neatly encapsulated in a single document, streamlining their remediation process.

Quantitative benefits have been a major highlight for our clients. In a recent case study, a healthcare provider reported a 45% increase in the speed of their vulnerability assessments, thanks to our bulk scanning capabilities. They no longer need to sift through large, unwieldy reports; instead, each domain's security posture is presented in a concise format. This efficiency gain has allowed them to reallocate resources to more critical tasks, enhancing their overall security posture.

Feedback from our users is invaluable and drives our continuous improvement efforts. One common request was more flexible report grouping, which prompted us to develop on-the-fly re-grouping capabilities. This feature allows users to dynamically reorganize PDF reports based on evolving security priorities or organizational changes. For instance, a user can now easily re-group PDFs by department rather than domain, adapting to their specific reporting needs. The implementation of this feature is straightforward:

def regroup_reports(reports, criterion):
    grouped_reports = {}
    for report in reports:
        key = report.get(criterion, 'Uncategorized')
        if key not in grouped_reports:
            grouped_reports[key] = []
        grouped_reports[key].append(report)
    return grouped_reports

# Example usage:
reports = fetch_reports()
grouped_by_department = regroup_reports(reports, 'department')

Limitations and Future Enhancements

While our current implementation of generating one PDF per domain in a bulk scan provides a streamlined method for organizing reports, it is not without its limitations. For instance, the system currently struggles with handling domains containing a large number of subdomains, leading to potential performance bottlenecks. Additionally, certain file types within the reports may not render accurately in PDF format, causing discrepancies in the final output. These issues highlight areas where improvements are not just beneficial but necessary for ensuring accuracy and efficiency in report generation.

Looking ahead, we have identified several potential enhancements that could significantly improve the scalability and functionality of our system. One proposed development is the integration of a more robust PDF generation library, such as PDFKit, to overcome current rendering issues. We also plan to introduce a feature that allows users to customize the level of detail included in each domain's report, offering a more tailored reporting experience. These improvements are aimed at making our platform more flexible and user-centric.

const generatePDF = async (domain) => {
  const pdf = new PDFDocument();
  const filePath = `/reports/${domain}.pdf`;
  pdf.pipe(fs.createWriteStream(filePath));
  pdf.text(`Report for ${domain}`);
  // Add more content to PDF here
  pdf.end();
};

We are eager to engage with the community to enhance these features further and address any additional concerns that users may encounter. Community feedback is invaluable, providing us with insights that can drive innovation and improve functionality. We invite developers and users alike to collaborate with us, whether by contributing code, suggesting new features, or reporting bugs. Together, we can refine our tool to better meet the needs of cybersecurity professionals everywhere.

• Scanning Behind 2FA: TOTP, OAuth Refresh, and the Authenticated Crawler
• Secret Scanner With Liveness: Distinguishing a Live AWS Key From a Revoked One
• The Session Watchdog: How We Re-Auth Mid-Scan Without You Noticing
• RuleSpec: How One Capability Matrix Drives 60+ Vuln Detectors