BOOST.Integrations

BOOST.Integrations is a multi-tenant email integration and processing system built with Apache Camel. It automatically converts incoming emails into business entities (Tasks, Work Orders, Documents) and manages file attachments with S3-compatible storage. The system provides a robust, scalable solution for integrating email workflows into the BOOST platform.

Getting Started

To understand how BOOST.Integrations works, we start with an overview of the main application entry point and how integrations are initialized.

Prerequisites

Before running BOOST.Integrations, ensure you have:

• Java 11+: The application requires Java 11 or higher
• Maven: For building the project
• MySQL Database: Access to the BOOST database infrastructure
• MinIO/S3: S3-compatible storage for file attachments
• JMS Broker: AMQP-compatible message broker (Apache Qpid)
• MQTT Broker: For real-time notifications

Running the Application

The main entry point is MainApp.java:

public class MainApp {
    public static void main(String[] args) throws Exception {
        BoostIntegrationsServer server = new BoostIntegrationsServer();
        server.start();
    }
}

The server initializes connection pools, loads organization configurations, and starts Camel contexts for each active integration.

Architecture Overview

BOOST.Integrations uses a multi-tenant, multi-context Apache Camel routing architecture:

MainApp
  └─ BoostIntegrationsServer
      ├─ Initialize HikariCP DataSource Pools (per organization)
      ├─ Load Organization Configuration from Master DB
      └─ Create Multiple CamelContext instances (one per integration)
          ├─ IMAP_PROCESSOR (Email Fetching)
          ├─ TASK_PROCESSOR (Email → Task conversion)
          ├─ DOCUMENT_PROCESSOR (Email → Document conversion)
          └─ WORKORDER_PROCESSOR (Email → WorkOrder conversion)

Key Components

• BoostIntegrationsServer: Central orchestrator managing all integrations
• Route Builders: Define Camel routes for each processor type
• Services: Business logic for entity creation and file management
• Secretary: JMS-based messaging for inter-service communication

Configuration

Application Properties

The main configuration file is config.properties:

# JMS Broker Configuration
broker_url=amqp://172.16.200.32:5672

# Redis Cache
redis_uri=redis://172.16.200.65:6379

# MQTT Broker
mqtt_host=tcp://p-s01-mqtt-01.westbahr.net:1883
mqtt_queue=sender.portal.messages

# MongoDB Backend
mongodb.uri=mongodb://172.16.200.35

# Message Queues
message_queue_name=boostv3.stageapi.messages
dispatcher_queue_name=boost.dispatcher.jobs

# Font Folder (for PDF processing)
font_folder=C:/Users/danie/git/BOOST.PDFBuilder/fonts

Database Configuration

HikariCP is used for connection pooling. The master database configuration is in hikari-qeeping-master.properties:

jdbcUrl=jdbc:mysql://10.245.10.35:3306/qeeping
dataSource.user=qeeping-master
dataSource.password=your-password
dataSource.minIdle=0
dataSource.maxLifetime=240000
dataSource.idleTimeout=240000

Integration Configuration

Integrations are stored in the SYS_BOOST_INTEGRATIONS table with the following properties:

• uuid: Unique identifier for the integration
• organizationUUID: The organization this integration belongs to
• processorType: Type of processor (IMAP_PROCESSOR, TASK_PROCESSOR, etc.)
• active: Whether the integration is active (1) or inactive (0)
• properties: JSON configuration specific to the processor type

Email Processing

IMAP Processor

The MailboxRouteBuilder handles email ingestion from IMAP servers:

from("imaps://{{mail.server}}?username={{mail.username}}&password={{mail.password}}")
    .process(new ExtractEmailDetailsProcessor())
    .bean(mailboxService, "storeEmail")
    .process(new HasEmailAttachmentsProcessor())
    .split(body())
        .bean(fileService, "save")
        .bean(thumbnailService, "createThumbnail")
    .end();

Properties for IMAP Configuration:

• mail.server: IMAP server hostname
• mail.username: Email account username
• mail.password: Email account password
• mail.protocol: Protocol (imaps for secure IMAP)
• mail.folder: Folder to monitor (default: INBOX)

Email Extraction

The ExtractEmailDetailsProcessor extracts:

• From/To/Subject: Email headers with MIME decoding support
• Text Body: Plain text content extracted recursively from multipart emails
• HTML Body: HTML content for rich text emails
• Attachments: File attachments with metadata

Email Storage

Emails are stored in the MAILBOX table with the following fields:

Field	Description
uuid	Unique identifier
fromAddress	Sender email address
toAddress	Recipient email address
subject	Email subject line
textBody	Plain text content
htmlBody	HTML content
receivedAt	Timestamp when received
status	Processing status (0=pending, 1=processing, 2=complete, -1=error)
entityId	Target entity type (Task, WorkOrder, Document)
createEntity	Flag to trigger entity creation

Entity Conversion

BOOST.Integrations can convert emails into three types of business entities.

Task Processor

The TaskRouteBuilder polls for emails marked for Task conversion:

from("timer:taskProcessor?period=60000")
    .bean(taskService, "processMailboxRecords");

Email to Task Mapping:

Email Field	Task Field
subject	title
textBody	description
fromAddress	createdBy (lookup)

TaskService creates the task and sends an MQTT notification:

public void processMailboxRecords(Exchange exchange) {
    // Fetch pending records
    List<MailboxRecord> records = fetchPendingRecords("Task");

    for (MailboxRecord record : records) {
        // Create task
        TasksRecord task = createTask(record);

        // Send MQTT notification
        secretary.sendCRUDMessage(organizationId, "Task", "CREATE", task);

        // Update status
        updateMailboxStatus(record, 2);
    }
}

WorkOrder Processor

The WorkOrderRouteBuilder follows the same pattern for Work Orders:

Email to WorkOrder Mapping:

Email Field	WorkOrder Field
subject	subject
textBody	body
fromAddress	reportedBy

Document Processor

The DocumentRouteBuilder creates documents with automatic numbering:

Email to Document Mapping:

Email Field	Document Field
subject	title
textBody	description
fromAddress	fromEmail

Document Numbering:

Documents receive sequential numbers from the DOCUMENT_NUMBERS table, ensuring unique document identifiers within each organization.

File Management

FileService

The FileService handles attachment storage:

public void save(Exchange exchange) {
    // Extract attachment from exchange
    Attachment attachment = exchange.getIn().getBody(Attachment.class);

    // Decode MIME-encoded filename
    String filename = MimeUtility.decodeText(attachment.getFilename());

    // Create database record
    FilesRecord file = createFileRecord(filename, attachment);

    // Generate MinIO object key
    String objectKey = file.getUuid() + "/" + filename;

    // Upload to MinIO
    minioService.uploadFile(objectKey, attachment.getInputStream());

    file.store();
}

FILES Table Schema:

Field	Description
uuid	Unique file identifier
filename	Original filename
mimeType	Content type
size	File size in bytes
objectKey	MinIO storage key
presignedUrl	Temporary access URL
thumbnailUrl	Thumbnail image URL
metadata	JSON metadata

MinIO Integration

Files are stored in MinIO (S3-compatible storage):

public class MinioService {
    private static final String ENDPOINT = "https://s3.se-01.westbahr.net";

    public String uploadFile(String objectKey, InputStream data) {
        minioClient.putObject(
            PutObjectArgs.builder()
                .bucket(bucketName)
                .object(objectKey)
                .stream(data, -1, 10485760)
                .build()
        );
        return generatePresignedUrl(objectKey);
    }

    public String generatePresignedUrl(String objectKey) {
        return minioClient.getPresignedObjectUrl(
            GetPresignedObjectUrlArgs.builder()
                .bucket(bucketName)
                .object(objectKey)
                .expiry(1, TimeUnit.MINUTES)
                .build()
        );
    }
}

Thumbnail Generation

The ThumbnailService generates thumbnails for images and PDFs:

Supported File Types:

• PDF (.pdf)
• JPEG (.jpg, .jpeg)
• PNG (.png)
• GIF (.gif)
• TIFF (.tif)
• BMP (.bmp)

Thumbnail requests are sent via JMS to the boost.v3.thumbnailgenerator queue:

public void createThumbnail(Exchange exchange) {
    FilesRecord file = exchange.getIn().getBody(FilesRecord.class);

    if (isThumbnailSupported(file.getFilename())) {
        String presignedUrl = minioService.generatePresignedUrl(file.getObjectKey());

        // Send to thumbnail generator
        JsonObject request = Json.createObjectBuilder()
            .add("fileUuid", file.getUuid())
            .add("url", presignedUrl)
            .build();

        String response = secretary.sendThumbnailRequest(request);

        // Update file with thumbnail URL
        file.setThumbnailUrl(parseThumbnailUrl(response));
        file.store();
    }
}

Messaging

Secretary

The Secretary class manages all JMS messaging:

public class Secretary {
    private ConnectionFactory connectionFactory;
    private JMSContext context;
    private JMSProducer mainDispatcher;
    private JMSProducer thumbnailProducer;

    public void sendCRUDMessage(String orgId, String entityType, String action, Object entity) {
        // Build MQTT topic
        String topic = "BOOST/" + md5Hash(orgId + SALT) + "/" + entityType;

        // Create dispatcher job
        JsonObject job = Json.createObjectBuilder()
            .add("jobType", "mqttCRUD")
            .add("topic", topic)
            .add("action", action)
            .add("body", serializeEntity(entity))
            .build();

        // Send to dispatcher queue
        mainDispatcher.send(dispatcherQueue, job.toString());
    }
}

Message Queues:

Queue	Purpose
boost.dispatcher.jobs	Main dispatcher for CRUD operations
boost.v3.thumbnailgenerator	Thumbnail generation requests

Job Types:

• documentAction: Document-specific operations
• mqttCRUD: MQTT-based CRUD notifications

MQTT Topics

MQTT topics follow the pattern: BOOST/{organizationHash}/{entityType}

The organization hash is an MD5 hash of the organization ID with a salt, providing tenant isolation.

Database Layer

jOOQ Integration

BOOST.Integrations uses jOOQ for type-safe database access. Generated classes provide:

• Tables: Strongly-typed table references
• Records: Active Record pattern with .store(), .fetch() methods
• DSLContext: Query builder with schema mapping

Multi-Tenant Database Access

Each organization has its own database, managed through DSLContext mapping:

public class DatabaseManager {
    private static Map<String, DSLContext> DSLMAP = new HashMap<>();
    private static Map<String, DataSource> ORGDSMAP = new HashMap<>();

    public DSLContext getDSLContext(String organizationUUID) {
        if (!DSLMAP.containsKey(organizationUUID)) {
            DataSource ds = getOrCreateDataSource(organizationUUID);

            Settings settings = new Settings()
                .withRenderMapping(new RenderMapping()
                    .withSchemata(new MappedSchema()
                        .withInput("portal_dev")
                        .withOutput(getSchemaName(organizationUUID))));

            DSLContext dsl = DSL.using(ds, SQLDialect.MYSQL, settings);
            DSLMAP.put(organizationUUID, dsl);
        }
        return DSLMAP.get(organizationUUID);
    }
}

Key Database Tables

Table	Purpose
SYS_BOOST_INTEGRATIONS	Integration configurations
MAILBOX	Email storage and processing status
FILES	File attachment metadata
TASKS	Task entities
WORK_ORDERS	Work order entities
DOCUMENTS	Document entities
DOCUMENT_NUMBERS	Sequential document numbering
ORGANIZATIONS	Organization/tenant definitions
DATASOURCES	Database connection configurations

Multi-Tenancy

BOOST.Integrations is designed for multi-tenant operation:

Tenant Isolation

• Separate Databases: Each organization has its own MySQL database
• Connection Pooling: HikariCP pools per organization via ORGDSMAP
• Schema Mapping: jOOQ RenderMapping translates schema references
• Camel Contexts: Isolated contexts per integration prevent cross-tenant interference

Organization Loading

Organizations are loaded from the master database with filtering:

public List<OrganizationsRecord> loadActiveOrganizations() {
    return masterDsl.selectFrom(ORGANIZATIONS)
        .where(ORGANIZATIONS.MASTER_DB.isNotNull())
        .and(ORGANIZATIONS.MIDDLEWARE_CLUSTER_UUID.eq(CLUSTER_UUID))
        .and(ORGANIZATIONS.DELETED_AT.isNull())
        .fetch();
}

Tenant-Scoped MQTT

MQTT topics include an organization hash for tenant isolation:

BOOST/{MD5(orgId + salt)}/Task
BOOST/{MD5(orgId + salt)}/WorkOrder
BOOST/{MD5(orgId + salt)}/Document

Data Flow

Email Ingestion Flow

IMAP Server
    ↓ (Camel Mail Component)
MailboxRouteBuilder
    ├─ ExtractEmailDetailsProcessor (parse headers, body, attachments)
    ├─ MailboxService.storeEmail() (write to MAILBOX table)
    ├─ HasEmailAttachmentsProcessor (detect attachments)
    ├─ [FOR EACH ATTACHMENT]
    │   ├─ FileService.save() (write to FILES table + upload to MinIO)
    │   └─ ThumbnailService.createThumbnail() (generate thumbnail)
    └─ MAILBOX record complete with STATUS=0

Entity Processing Flow

Timer (60 seconds)
    ↓
Entity Processor (Task/WorkOrder/Document)
    ├─ Query MAILBOX where STATUS=0 and ENTITYID={type}
    ├─ Update STATUS=1 (processing)
    ├─ Create entity record in respective table
    ├─ Send MQTT CRUD notification via Secretary
    └─ Update STATUS=2 (complete) or STATUS=-1 (error)

Extension Points

Adding New Processor Types

1. Create a new RouteBuilder class:

public class CustomRouteBuilder extends RouteBuilder {
    @Override
    public void configure() {
        from("timer:customProcessor?period=60000")
            .bean(customService, "process");
    }
}

2. Register in BoostIntegrationsServer:

case "CUSTOM_PROCESSOR":
    context.addRoutes(new CustomRouteBuilder(dsl, secretary, properties));
    break;

3. Add integration record to SYS_BOOST_INTEGRATIONS

Adding New Entity Types

1. Create a Service class following the pattern:

public class CustomEntityService {
    public void processMailboxRecords(Exchange exchange) {
        List<MailboxRecord> records = fetchPendingRecords("CustomEntity");
        for (MailboxRecord record : records) {
            CustomRecord entity = createEntity(record);
            secretary.sendCRUDMessage(orgId, "CustomEntity", "CREATE", entity);
            updateMailboxStatus(record, 2);
        }
    }
}

2. Create the corresponding RouteBuilder
3. Register the processor type in the server

Custom Processors

Implement Camel's Processor interface for custom processing logic:

public class CustomProcessor implements Processor {
    @Override
    public void process(Exchange exchange) throws Exception {
        // Custom processing logic
        Message message = exchange.getIn();
        // Transform, validate, enrich...
        exchange.getMessage().setBody(transformedData);
    }
}

Dependencies

Core Frameworks

Dependency	Version	Purpose
Apache Camel	4.10.0	EIP-based routing and integration
jOOQ	3.19.6	Type-safe SQL and ORM
HikariCP	6.2.1	High-performance connection pooling

Data Access

Dependency	Version	Purpose
MySQL Connector	9.2.0	MySQL database driver
MinIO	8.5.17	S3-compatible object storage

Messaging

Dependency	Version	Purpose
Apache Qpid JMS	2.5.0	JMS 3.1 AMQP client
Jakarta Mail	2.0.1	IMAP email client

Utilities

Dependency	Version	Purpose
Apache PDFBox	3.0.4	PDF text extraction
SLF4J/Reload4j	-	Logging framework

Troubleshooting

Common Issues

Email Connection Failures:

• Verify IMAP server credentials in integration properties
• Check firewall rules for IMAP ports (993 for IMAPS)
• Ensure SSL certificates are valid

Entity Creation Failures:

• Check MAILBOX table for records with STATUS=-1
• Review application logs for exception details
• Verify database connectivity for the target organization

File Upload Failures:

• Verify MinIO endpoint accessibility
• Check bucket permissions and credentials
• Review file size limits

Logging

Logging is configured in log4j.properties:

log4j.rootLogger=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L - %m%n

For more verbose logging, set specific packages to DEBUG:

log4j.logger.com.luqon.boost.integrations=DEBUG
log4j.logger.org.apache.camel=DEBUG

Security Considerations

Credentials Management

• Store sensitive credentials in environment variables or secure vaults
• Avoid hardcoding credentials in source code
• Use encrypted connections for all external services

Multi-Tenant Security

• Organization databases are isolated by design
• MQTT topics use hashed organization IDs
• Schema mapping prevents cross-tenant data access

File Storage Security

• MinIO presigned URLs have short expiry times
• Files are stored with organization-scoped object keys
• Access requires valid presigned URLs