OverView DropPoint

S3-compatible file drop-off, with dedicated buckets per organization

OverView DropPoint is an S3-compatible file drop-off point. Each organization has its own buckets, reached with its own credentials.

Files can be uploaded from the OverView web interface, with the AWS CLI, or with an S3 SDK. That compatibility is what makes automated uploads possible, such as a scheduled daily export. S3 here refers to the protocol: DropPoint is not tied to AWS, and files are stored in France, with a French hosting provider.

Overview

DropPoint provides:

  • Complete isolation between organizations
  • AWS CLI compatibility for easy integration
  • Dedicated S3 buckets per connector type
  • Hosted in France, with a French hosting provider, not tied to AWS

Generating Credentials

Steps to Generate Your Credentials

  1. In the OverView application, navigate to Settings > Security > S3 Credentials
  2. Click the + button to generate new credentials
  3. Your credentials are displayed:
    • Access Key ID: Unique identifier for your account
    • Secret Access Key: Secret key (keep it confidential)
    • S3 Endpoint: https://s3.oversoc.com

Credential Security

Caution
Important
  • The Secret Access Key is only visible at generation time
  • Copy and save your credentials immediately
  • Never share your credentials with third parties

Choosing a Client

DropPoint speaks the S3 API, so any S3 client works once it is pointed at https://s3.oversoc.com. These are the ones covered here:

ClientPlatformsSuited to
AWS CLIWindows, macOS, LinuxThe recommended starting point: one tool, same commands everywhere, easy to script
AWS Tools for PowerShellWindowsWindows scripting without installing the AWS CLI, using native cmdlets
rcloneWindows, macOS, LinuxMirroring a folder, since it uploads only what changed
Python SDK (boto3)Windows, macOS, LinuxUploading from an existing script or application

A graphical client also works for occasional manual uploads. Cyberduck (Windows, macOS) and WinSCP (Windows) both accept a custom S3 endpoint. Create a profile of type S3, set the server to s3.oversoc.com, and use your Access Key ID and Secret Access Key as username and password.

Caution
Path style addressing is required

DropPoint reads the bucket from the path, as in https://s3.oversoc.com/your-bucket/export.csv. The bucket is never part of the hostname, so virtual hosted addressing (your-bucket.s3.oversoc.com) does not work. The clients below are configured accordingly, and a graphical client needs its "path style" option enabled.

AWS CLI Configuration

Install the AWS CLI

# Windows (PowerShell)
winget install --id Amazon.AWSCLI
# macOS
brew install awscli
# Linux
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip
unzip awscliv2.zip && sudo ./aws/install

Configure the profile

Configure your AWS CLI profile using the credentials generated above:

aws configure set aws_access_key_id YOUR_ACCESS_KEY --profile ovs_your_org
aws configure set aws_secret_access_key YOUR_SECRET_KEY --profile ovs_your_org
aws configure set region fr-par --profile ovs_your_org
aws configure set endpoint_url https://s3.oversoc.com --profile ovs_your_org
aws configure set s3.addressing_style path --profile ovs_your_org

The same five commands work on Windows, macOS and Linux. s3.addressing_style defaults to auto, which tries to put the bucket in the hostname: setting it to path keeps requests in the form DropPoint accepts. endpoint_url in the profile is what lets the later commands omit --endpoint-url every time. It requires AWS CLI v2.13 or later; on an older version, pass --endpoint-url https://s3.oversoc.com on each command instead.

Check that the profile works:

aws s3 ls --profile ovs_your_org

The command lists your organization's buckets. An empty output means the profile reached DropPoint but no bucket exists yet.

Using S3 Buckets

Bucket Structure

Each organization has dedicated S3 buckets for each connector. Buckets are named directly after the connector type:

  • azure - For Azure data
  • sentinelone - For SentinelOne data
  • vmware - For VMware data
  • And other connectors based on your needs

Files go at the root of the bucket, in CSV or JSON. OverView detects new files and imports them on the next synchronization. Each connector's page names the bucket it reads.

List Your Available Buckets

# List all available buckets for your organization
aws s3 ls --profile ovs_your_org

Uploading Files

Via AWS CLI

# List contents of a specific bucket
aws s3 ls s3://your-bucket/ --profile ovs_your_org

# Upload a file to the bucket
aws s3 cp file.json s3://your-bucket/file.json --profile ovs_your_org

# Upload a whole folder
aws s3 sync ./exports/ s3://your-bucket/ --profile ovs_your_org

# Download a file from a bucket
aws s3 cp s3://your-bucket/file.json ./file.json --profile ovs_your_org

Keep the listing in a script: it is what tells an unattended run whether the upload landed.

Via AWS Tools for PowerShell (Windows)

The AWS Tools for PowerShell give Windows native cmdlets, with no AWS CLI to install. Install the S3 module once:

Install-Module -Name AWS.Tools.Installer -Repository PSGallery -Scope CurrentUser
Install-AWSToolsModule -Name AWS.Tools.S3 -Scope CurrentUser

Store the credentials as a named profile, then set the region for the session:

Set-AWSCredential -AccessKey YOUR_ACCESS_KEY -SecretKey YOUR_SECRET_KEY -StoreAs ovs_your_org
Set-DefaultAWSRegion -Region fr-par

List the buckets and upload a file:

Get-S3Bucket -ProfileName ovs_your_org -EndpointUrl https://s3.oversoc.com -Region fr-par

Write-S3Object -BucketName your-bucket -Key export.csv -File .\export.csv `
  -ProfileName ovs_your_org -EndpointUrl https://s3.oversoc.com -Region fr-par
Note
Why -Region on every cmdlet

Set-AWSCredential -StoreAs stores the keys but no region, and the SDK needs one to sign a request. Set-DefaultAWSRegion covers the current session, while -Region on each cmdlet also covers a script run from Task Scheduler, where no session default has been set.

Path style addressing needs no extra flag: the cmdlets switch to it on their own as soon as -EndpointUrl is supplied.

Via rclone

rclone compares source and destination, so a scheduled run uploads only what changed. Add a remote to rclone.conf:

[droppoint]
type = s3
provider = Other
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
endpoint = https://s3.oversoc.com
region = fr-par
force_path_style = true

force_path_style is not optional here, for the reason given above.

The config file lives at %APPDATA%\rclone\rclone.conf on Windows and ~/.config/rclone/rclone.conf on macOS and Linux. rclone config file prints the path in use.

# List the buckets
rclone lsd droppoint:

# Copy one file
rclone copy ./export.csv droppoint:your-bucket/

# Mirror a folder, uploading only what changed
rclone sync ./exports/ droppoint:your-bucket/
Caution
sync deletes

rclone sync makes the destination match the source, which means removing files from the bucket that no longer exist locally. Use rclone copy to add without ever deleting, and try a run with --dry-run first.

Via Python SDK (boto3)

import boto3

# Configure S3 client
session = boto3.Session(
    aws_access_key_id='YOUR_ACCESS_KEY',
    aws_secret_access_key='YOUR_SECRET_KEY',
    region_name='fr-par'
)

s3_client = session.client(
    's3',
    endpoint_url='https://s3.oversoc.com'
)

# List all available buckets
response = s3_client.list_buckets()
for bucket in response['Buckets']:
    print(f"Bucket: {bucket['Name']}")

# Upload a file to the Azure bucket
s3_client.upload_file(
    'local_file.json',
    'your-bucket',
    'remote_file.json'
)

# List objects in a specific bucket
response = s3_client.list_objects_v2(Bucket='your-bucket')
for obj in response.get('Contents', []):
    print(f"File: {obj['Key']}, Size: {obj['Size']} bytes")

Automating Uploads

Pair an upload with a scheduled task so exports reach OverView without anyone logging in. Each upload triggers a new import, so a daily export keeps the inventory current.

Replace your-bucket with the connector's bucket name, shown on that connector's own documentation page.

Linux

# /etc/cron.d/overview-export : every day at 02:30
30 2 * * * root /usr/local/bin/aws s3 cp /var/exports/export.csv s3://your-bucket/export.csv --profile ovs_your_org

cron runs with a minimal environment, so give aws its full path. The profile is read from the home directory of the user the job runs as, root above.

macOS

cron works the same way through crontab -e. launchd is the supported scheduler on macOS and survives reboots more predictably:

<!-- ~/Library/LaunchAgents/com.oversoc.export.plist -->
<?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0"><dict>
  <key>Label</key><string>com.oversoc.export</string>
  <key>ProgramArguments</key>
  <array>
    <string>/opt/homebrew/bin/aws</string><string>s3</string><string>cp</string>
    <string>/Users/you/exports/export.csv</string>
    <string>s3://your-bucket/export.csv</string>
    <string>--profile</string><string>ovs_your_org</string>
  </array>
  <key>StartCalendarInterval</key><dict><key>Hour</key><integer>2</integer><key>Minute</key><integer>30</integer></dict>
</dict></plist>

Load it with launchctl load ~/Library/LaunchAgents/com.oversoc.export.plist.

Windows

Save the upload as a script, so the scheduled task holds one command:

# C:\Scripts\Send-OverViewExport.ps1
$ErrorActionPreference = 'Stop'
Write-S3Object -BucketName your-bucket -Key export.csv -File 'C:\Exports\export.csv' `
  -ProfileName ovs_your_org -EndpointUrl https://s3.oversoc.com -Region fr-par

Register it with Task Scheduler:

$action  = New-ScheduledTaskAction -Execute 'powershell.exe' `
  -Argument '-NoProfile -ExecutionPolicy Bypass -File C:\Scripts\Send-OverViewExport.ps1'
$trigger = New-ScheduledTaskTrigger -Daily -At 02:30
Register-ScheduledTask -TaskName 'OverView export' -Action $action -Trigger $trigger

The task runs as the registered user, and Set-AWSCredential stores the profile per user. Register the task under the same account that stored the credentials, or the script will not find the profile.

Troubleshooting

Common Errors

Every client surfaces the error code DropPoint returns. The code is what tells the causes apart:

ErrorWhat it meansWhat to check
InvalidAccessKeyIdThe Access Key ID is unknownA typo in the key, or credentials that were revoked. Generate new ones from Settings > Security > S3 Credentials
SignatureDoesNotMatchThe signature does not matchUsually a wrong Secret Access Key. Also happens when the machine clock is off by more than about 15 minutes, since the signature covers the request timestamp: check NTP
AccessDeniedAuthenticated, but not allowed hereThe bucket is a connector these credentials do not cover. Confirm the bucket name against the connector's page, then contact OverSOC support to have the connector added to your credentials
AccessDenied, "Credentials are inactive"The credentials were disabledGenerate new credentials, or contact OverSOC support
AccessDenied, "Missing authorization"The request carried no signatureThe client is not configured for S3 signature v4, or is sending anonymous requests
EntityTooLargeOne request went over 500 MBUpload larger files in parts. aws s3 cp and rclone do this on their own; a hand-written SDK call may need multipart enabling. Splitting the export also works
NoSuchKeyThe object does not existThe key is spelled differently, or the file went into another bucket
InvalidBucketNameThe bucket name is not acceptedThe name must be the connector identifier, lowercase, as shown on the connector's page
XAmzContentSHA256MismatchThe body does not match its checksumA proxy is altering the request. Exclude s3.oversoc.com from any rewriting or inspecting proxy

Where to read the error

# The CLI prints the code on failure; --debug shows the full exchange
aws s3 cp ./export.csv s3://your-bucket/export.csv --profile ovs_your_org --debug

Support

For assistance, contact support@oversoc.com.

Limits and Quotas

  • Maximum size per request: 500 MB. A larger file still goes through, in parts: aws s3 cp and rclone switch to a multipart upload on their own, and each part stays under that limit. Over it, a single request is rejected with EntityTooLarge.
  • Number of files and total volume: no per-organization quota is enforced today.