Skip to content

BOOST.Integrations ​

BOOST.Integrations is a multi-tenant email integration and processing system built with Apache Camel. It automatically converts incoming emails into business entities (Tasks, Work Orders, Documents) and manages file attachments with S3-compatible storage. The system provides a robust, scalable solution for integrating email workflows into the BOOST platform.

Getting Started ​

To understand how BOOST.Integrations works, we start with an overview of the main application entry point and how integrations are initialized.

Prerequisites ​

Before running BOOST.Integrations, ensure you have:

  • Java 11+: The application requires Java 11 or higher
  • Maven: For building the project
  • MySQL Database: Access to the BOOST database infrastructure
  • MinIO/S3: S3-compatible storage for file attachments
  • JMS Broker: AMQP-compatible message broker (Apache Qpid)
  • MQTT Broker: For real-time notifications

Running the Application ​

The main entry point is MainApp.java:

java
public class MainApp {
    public static void main(String[] args) throws Exception {
        BoostIntegrationsServer server = new BoostIntegrationsServer();
        server.start();
    }
}

The server initializes connection pools, loads organization configurations, and starts Camel contexts for each active integration.

Architecture Overview ​

BOOST.Integrations uses a multi-tenant, multi-context Apache Camel routing architecture:

MainApp
  └─ BoostIntegrationsServer
      ├─ Initialize HikariCP DataSource Pools (per organization)
      ├─ Load Organization Configuration from Master DB
      └─ Create Multiple CamelContext instances (one per integration)
          ├─ IMAP_PROCESSOR (Email Fetching)
          ├─ TASK_PROCESSOR (Email → Task conversion)
          ├─ DOCUMENT_PROCESSOR (Email → Document conversion)
          └─ WORKORDER_PROCESSOR (Email → WorkOrder conversion)

Key Components ​

  • BoostIntegrationsServer: Central orchestrator managing all integrations
  • Route Builders: Define Camel routes for each processor type
  • Services: Business logic for entity creation and file management
  • Secretary: JMS-based messaging for inter-service communication

Configuration ​

Application Properties ​

The main configuration file is config.properties:

properties
# JMS Broker Configuration
broker_url=amqp://172.16.200.32:5672

# Redis Cache
redis_uri=redis://172.16.200.65:6379

# MQTT Broker
mqtt_host=tcp://p-s01-mqtt-01.westbahr.net:1883
mqtt_queue=sender.portal.messages

# MongoDB Backend
mongodb.uri=mongodb://172.16.200.35

# Message Queues
message_queue_name=boostv3.stageapi.messages
dispatcher_queue_name=boost.dispatcher.jobs

# Font Folder (for PDF processing)
font_folder=C:/Users/danie/git/BOOST.PDFBuilder/fonts

Database Configuration ​

HikariCP is used for connection pooling. The master database configuration is in hikari-qeeping-master.properties:

properties
jdbcUrl=jdbc:mysql://10.245.10.35:3306/qeeping
dataSource.user=qeeping-master
dataSource.password=your-password
dataSource.minIdle=0
dataSource.maxLifetime=240000
dataSource.idleTimeout=240000

Integration Configuration ​

Integrations are stored in the SYS_BOOST_INTEGRATIONS table with the following properties:

  • uuid: Unique identifier for the integration
  • organizationUUID: The organization this integration belongs to
  • processorType: Type of processor (IMAP_PROCESSOR, TASK_PROCESSOR, etc.)
  • active: Whether the integration is active (1) or inactive (0)
  • properties: JSON configuration specific to the processor type

Email Processing ​

IMAP Processor ​

The MailboxRouteBuilder handles email ingestion from IMAP servers:

java
from("imaps://{{mail.server}}?username={{mail.username}}&password={{mail.password}}")
    .process(new ExtractEmailDetailsProcessor())
    .bean(mailboxService, "storeEmail")
    .process(new HasEmailAttachmentsProcessor())
    .split(body())
        .bean(fileService, "save")
        .bean(thumbnailService, "createThumbnail")
    .end();

Properties for IMAP Configuration:

  • mail.server: IMAP server hostname
  • mail.username: Email account username
  • mail.password: Email account password
  • mail.protocol: Protocol (imaps for secure IMAP)
  • mail.folder: Folder to monitor (default: INBOX)

Email Extraction ​

The ExtractEmailDetailsProcessor extracts:

  • From/To/Subject: Email headers with MIME decoding support
  • Text Body: Plain text content extracted recursively from multipart emails
  • HTML Body: HTML content for rich text emails
  • Attachments: File attachments with metadata

Email Storage ​

Emails are stored in the MAILBOX table with the following fields:

FieldDescription
uuidUnique identifier
fromAddressSender email address
toAddressRecipient email address
subjectEmail subject line
textBodyPlain text content
htmlBodyHTML content
receivedAtTimestamp when received
statusProcessing status (0=pending, 1=processing, 2=complete, -1=error)
entityIdTarget entity type (Task, WorkOrder, Document)
createEntityFlag to trigger entity creation

Entity Conversion ​

BOOST.Integrations can convert emails into three types of business entities.

Task Processor ​

The TaskRouteBuilder polls for emails marked for Task conversion:

java
from("timer:taskProcessor?period=60000")
    .bean(taskService, "processMailboxRecords");

Email to Task Mapping:

Email FieldTask Field
subjecttitle
textBodydescription
fromAddresscreatedBy (lookup)

TaskService creates the task and sends an MQTT notification:

java
public void processMailboxRecords(Exchange exchange) {
    // Fetch pending records
    List<MailboxRecord> records = fetchPendingRecords("Task");

    for (MailboxRecord record : records) {
        // Create task
        TasksRecord task = createTask(record);

        // Send MQTT notification
        secretary.sendCRUDMessage(organizationId, "Task", "CREATE", task);

        // Update status
        updateMailboxStatus(record, 2);
    }
}

WorkOrder Processor ​

The WorkOrderRouteBuilder follows the same pattern for Work Orders:

Email to WorkOrder Mapping:

Email FieldWorkOrder Field
subjectsubject
textBodybody
fromAddressreportedBy

Document Processor ​

The DocumentRouteBuilder creates documents with automatic numbering:

Email to Document Mapping:

Email FieldDocument Field
subjecttitle
textBodydescription
fromAddressfromEmail

Document Numbering:

Documents receive sequential numbers from the DOCUMENT_NUMBERS table, ensuring unique document identifiers within each organization.

File Management ​

FileService ​

The FileService handles attachment storage:

java
public void save(Exchange exchange) {
    // Extract attachment from exchange
    Attachment attachment = exchange.getIn().getBody(Attachment.class);

    // Decode MIME-encoded filename
    String filename = MimeUtility.decodeText(attachment.getFilename());

    // Create database record
    FilesRecord file = createFileRecord(filename, attachment);

    // Generate MinIO object key
    String objectKey = file.getUuid() + "/" + filename;

    // Upload to MinIO
    minioService.uploadFile(objectKey, attachment.getInputStream());

    file.store();
}

FILES Table Schema:

FieldDescription
uuidUnique file identifier
filenameOriginal filename
mimeTypeContent type
sizeFile size in bytes
objectKeyMinIO storage key
presignedUrlTemporary access URL
thumbnailUrlThumbnail image URL
metadataJSON metadata

MinIO Integration ​

Files are stored in MinIO (S3-compatible storage):

java
public class MinioService {
    private static final String ENDPOINT = "https://s3.se-01.westbahr.net";

    public String uploadFile(String objectKey, InputStream data) {
        minioClient.putObject(
            PutObjectArgs.builder()
                .bucket(bucketName)
                .object(objectKey)
                .stream(data, -1, 10485760)
                .build()
        );
        return generatePresignedUrl(objectKey);
    }

    public String generatePresignedUrl(String objectKey) {
        return minioClient.getPresignedObjectUrl(
            GetPresignedObjectUrlArgs.builder()
                .bucket(bucketName)
                .object(objectKey)
                .expiry(1, TimeUnit.MINUTES)
                .build()
        );
    }
}

Thumbnail Generation ​

The ThumbnailService generates thumbnails for images and PDFs:

Supported File Types:

  • PDF (.pdf)
  • JPEG (.jpg, .jpeg)
  • PNG (.png)
  • GIF (.gif)
  • TIFF (.tif)
  • BMP (.bmp)

Thumbnail requests are sent via JMS to the boost.v3.thumbnailgenerator queue:

java
public void createThumbnail(Exchange exchange) {
    FilesRecord file = exchange.getIn().getBody(FilesRecord.class);

    if (isThumbnailSupported(file.getFilename())) {
        String presignedUrl = minioService.generatePresignedUrl(file.getObjectKey());

        // Send to thumbnail generator
        JsonObject request = Json.createObjectBuilder()
            .add("fileUuid", file.getUuid())
            .add("url", presignedUrl)
            .build();

        String response = secretary.sendThumbnailRequest(request);

        // Update file with thumbnail URL
        file.setThumbnailUrl(parseThumbnailUrl(response));
        file.store();
    }
}

Messaging ​

Secretary ​

The Secretary class manages all JMS messaging:

java
public class Secretary {
    private ConnectionFactory connectionFactory;
    private JMSContext context;
    private JMSProducer mainDispatcher;
    private JMSProducer thumbnailProducer;

    public void sendCRUDMessage(String orgId, String entityType, String action, Object entity) {
        // Build MQTT topic
        String topic = "BOOST/" + md5Hash(orgId + SALT) + "/" + entityType;

        // Create dispatcher job
        JsonObject job = Json.createObjectBuilder()
            .add("jobType", "mqttCRUD")
            .add("topic", topic)
            .add("action", action)
            .add("body", serializeEntity(entity))
            .build();

        // Send to dispatcher queue
        mainDispatcher.send(dispatcherQueue, job.toString());
    }
}

Message Queues:

QueuePurpose
boost.dispatcher.jobsMain dispatcher for CRUD operations
boost.v3.thumbnailgeneratorThumbnail generation requests

Job Types:

  • documentAction: Document-specific operations
  • mqttCRUD: MQTT-based CRUD notifications

MQTT Topics ​

MQTT topics follow the pattern: BOOST/{organizationHash}/{entityType}

The organization hash is an MD5 hash of the organization ID with a salt, providing tenant isolation.

Database Layer ​

jOOQ Integration ​

BOOST.Integrations uses jOOQ for type-safe database access. Generated classes provide:

  • Tables: Strongly-typed table references
  • Records: Active Record pattern with .store(), .fetch() methods
  • DSLContext: Query builder with schema mapping

Multi-Tenant Database Access ​

Each organization has its own database, managed through DSLContext mapping:

java
public class DatabaseManager {
    private static Map<String, DSLContext> DSLMAP = new HashMap<>();
    private static Map<String, DataSource> ORGDSMAP = new HashMap<>();

    public DSLContext getDSLContext(String organizationUUID) {
        if (!DSLMAP.containsKey(organizationUUID)) {
            DataSource ds = getOrCreateDataSource(organizationUUID);

            Settings settings = new Settings()
                .withRenderMapping(new RenderMapping()
                    .withSchemata(new MappedSchema()
                        .withInput("portal_dev")
                        .withOutput(getSchemaName(organizationUUID))));

            DSLContext dsl = DSL.using(ds, SQLDialect.MYSQL, settings);
            DSLMAP.put(organizationUUID, dsl);
        }
        return DSLMAP.get(organizationUUID);
    }
}

Key Database Tables ​

TablePurpose
SYS_BOOST_INTEGRATIONSIntegration configurations
MAILBOXEmail storage and processing status
FILESFile attachment metadata
TASKSTask entities
WORK_ORDERSWork order entities
DOCUMENTSDocument entities
DOCUMENT_NUMBERSSequential document numbering
ORGANIZATIONSOrganization/tenant definitions
DATASOURCESDatabase connection configurations

Multi-Tenancy ​

BOOST.Integrations is designed for multi-tenant operation:

Tenant Isolation ​

  • Separate Databases: Each organization has its own MySQL database
  • Connection Pooling: HikariCP pools per organization via ORGDSMAP
  • Schema Mapping: jOOQ RenderMapping translates schema references
  • Camel Contexts: Isolated contexts per integration prevent cross-tenant interference

Organization Loading ​

Organizations are loaded from the master database with filtering:

java
public List<OrganizationsRecord> loadActiveOrganizations() {
    return masterDsl.selectFrom(ORGANIZATIONS)
        .where(ORGANIZATIONS.MASTER_DB.isNotNull())
        .and(ORGANIZATIONS.MIDDLEWARE_CLUSTER_UUID.eq(CLUSTER_UUID))
        .and(ORGANIZATIONS.DELETED_AT.isNull())
        .fetch();
}

Tenant-Scoped MQTT ​

MQTT topics include an organization hash for tenant isolation:

BOOST/{MD5(orgId + salt)}/Task
BOOST/{MD5(orgId + salt)}/WorkOrder
BOOST/{MD5(orgId + salt)}/Document

Data Flow ​

Email Ingestion Flow ​

IMAP Server
    ↓ (Camel Mail Component)
MailboxRouteBuilder
    ├─ ExtractEmailDetailsProcessor (parse headers, body, attachments)
    ├─ MailboxService.storeEmail() (write to MAILBOX table)
    ├─ HasEmailAttachmentsProcessor (detect attachments)
    ├─ [FOR EACH ATTACHMENT]
    │   ├─ FileService.save() (write to FILES table + upload to MinIO)
    │   └─ ThumbnailService.createThumbnail() (generate thumbnail)
    └─ MAILBOX record complete with STATUS=0

Entity Processing Flow ​

Timer (60 seconds)
    ↓
Entity Processor (Task/WorkOrder/Document)
    ├─ Query MAILBOX where STATUS=0 and ENTITYID={type}
    ├─ Update STATUS=1 (processing)
    ├─ Create entity record in respective table
    ├─ Send MQTT CRUD notification via Secretary
    └─ Update STATUS=2 (complete) or STATUS=-1 (error)

Extension Points ​

Adding New Processor Types ​

  1. Create a new RouteBuilder class:
java
public class CustomRouteBuilder extends RouteBuilder {
    @Override
    public void configure() {
        from("timer:customProcessor?period=60000")
            .bean(customService, "process");
    }
}
  1. Register in BoostIntegrationsServer:
java
case "CUSTOM_PROCESSOR":
    context.addRoutes(new CustomRouteBuilder(dsl, secretary, properties));
    break;
  1. Add integration record to SYS_BOOST_INTEGRATIONS

Adding New Entity Types ​

  1. Create a Service class following the pattern:
java
public class CustomEntityService {
    public void processMailboxRecords(Exchange exchange) {
        List<MailboxRecord> records = fetchPendingRecords("CustomEntity");
        for (MailboxRecord record : records) {
            CustomRecord entity = createEntity(record);
            secretary.sendCRUDMessage(orgId, "CustomEntity", "CREATE", entity);
            updateMailboxStatus(record, 2);
        }
    }
}
  1. Create the corresponding RouteBuilder
  2. Register the processor type in the server

Custom Processors ​

Implement Camel's Processor interface for custom processing logic:

java
public class CustomProcessor implements Processor {
    @Override
    public void process(Exchange exchange) throws Exception {
        // Custom processing logic
        Message message = exchange.getIn();
        // Transform, validate, enrich...
        exchange.getMessage().setBody(transformedData);
    }
}

Dependencies ​

Core Frameworks ​

DependencyVersionPurpose
Apache Camel4.10.0EIP-based routing and integration
jOOQ3.19.6Type-safe SQL and ORM
HikariCP6.2.1High-performance connection pooling

Data Access ​

DependencyVersionPurpose
MySQL Connector9.2.0MySQL database driver
MinIO8.5.17S3-compatible object storage

Messaging ​

DependencyVersionPurpose
Apache Qpid JMS2.5.0JMS 3.1 AMQP client
Jakarta Mail2.0.1IMAP email client

Utilities ​

DependencyVersionPurpose
Apache PDFBox3.0.4PDF text extraction
SLF4J/Reload4j-Logging framework

Troubleshooting ​

Common Issues ​

Email Connection Failures:

  • Verify IMAP server credentials in integration properties
  • Check firewall rules for IMAP ports (993 for IMAPS)
  • Ensure SSL certificates are valid

Entity Creation Failures:

  • Check MAILBOX table for records with STATUS=-1
  • Review application logs for exception details
  • Verify database connectivity for the target organization

File Upload Failures:

  • Verify MinIO endpoint accessibility
  • Check bucket permissions and credentials
  • Review file size limits

Logging ​

Logging is configured in log4j.properties:

properties
log4j.rootLogger=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L - %m%n

For more verbose logging, set specific packages to DEBUG:

properties
log4j.logger.com.luqon.boost.integrations=DEBUG
log4j.logger.org.apache.camel=DEBUG

Security Considerations ​

Credentials Management ​

  • Store sensitive credentials in environment variables or secure vaults
  • Avoid hardcoding credentials in source code
  • Use encrypted connections for all external services

Multi-Tenant Security ​

  • Organization databases are isolated by design
  • MQTT topics use hashed organization IDs
  • Schema mapping prevents cross-tenant data access

File Storage Security ​

  • MinIO presigned URLs have short expiry times
  • Files are stored with organization-scoped object keys
  • Access requires valid presigned URLs