Administration
Scalability
As regards scalability, BitSwan is designed to meet the most extreme requirements. The BitSwan product units are set in Docker containers, which enables to make use of unlimited possibilities of BitSwan optimization according to the number of data processed. This for instance means the possibility of horizontal system scaling by adding a computing junction as well as vertical scaling by increasing HW means in one junction.
Because of internal architecture, other junctions can be promptly added so as to unlimitedly increase computing performance during data processing. Container architecture makes it possible for users to install BitSwan on their own hardware, private cloud (virtual device) or a public cloud like Microsoft Azure, Amazon, Google Cloud etc. BitSwan enables combining infrastructure items according to user requirements and supports orchestral tools like Kubernetes, Docker Swarm and Docker Composer to widen operational capabilities of your infrastructure.
Throughput
The system is designed to enable extreme-speed data processing. Typical performance amounts to 5 000 messages per second (EPS) per one CPU core. This means that on a common eight-processor server it is possible to reach throughput as high as 40 000 messages per second. Thus in a cluster with three such servers the speed of 120 000 messages per second can be reached.
In case it is necessary to dispatch a larger number of incoming events promptly, messages are put in a co called queue from which they are gradually fetched to be processed. This completely eliminates loss of any incoming data – they are only processed with some delay. This delay (time drift) is monitored and administrators are warned about it. The BitSwan product implicitly measures data throughput via data pumps. It is further possible to be tracked and monitored in a Grafana-type tool.
It is possible to monitor the number of incoming events, outcoming events and junk events together with other metrics which can arbitrarily be set for a concrete solution by users.
Clustering
The BitSwan product data pumps can be triggered in more instances at various junctions within a corporate infrastructure. Each instance makes use of one CPU core - so they can be arbitrarily scaled by adding new container instances. Because of internal mechanisms, a trouble-free data processing without any duplicities can be ensured.
Clustering is implicitly supported by ElasticSearch tool which can also run at more junctions for the purpose of increasing computing performance.
High accessibility
The above mentioned qualities also enable to run the whole system in a high accessibility regime. In such a case it is necessary to increase hardware resources correspondingly or combine hardware resources with virtual ones. Thus the system can be run in two or more geographically separated localities (e.g. other data centres) as well. Latency requirement among such data centres is 1 ms.
Cluster configuration
Basic cluster junction:
| Component | Specification |
|---|---|
| CPU | AMD Ryzen 7 2700 8/16T-Core, 3.7GHz, 105W, 16MB L3 cache |
| RAM | 4x 16GB DDR4 2666 |
| SDD | 128GB, M.2 pro OS a swap |
| NIC | Gigabit Ethernet port |
| Physical | Tower or 4U Rack-mounted |
Local storage:
Each cluster junction is necessary to be equipped with one of the following disk store variants:
- 2x 3TB HDD 3.5”, SATA III, 256MB cache, 5400 revolutions
- 6x 14TB HDD 3.5”, SATA III, 256MB cache, 7200 revolutions
- 6x 4TB SSD, QVO, SATA III, random entry 89 kIOPS, random reading 97 kIOPS
It is not recommended to combine fast (SSD) and slow (HDD) technologies at one junction – on the contrary combining so called fast junctions (i.e. equipped with SDD) and slow junctins (with HDD) is recommended. Slow junctions are then used for archiving data which are not actively used – for instance historically older entries.
Recommended cluster compilations:
1 junction
- ● Testing
- ● Pre-production operation
- ● Low production operation without HA
3 junctions
- ● The smallest recommended production cluster
- ● Provides resistance to one junction failure
2x3 junctions
- Configuration suitable for High-Availability configuration
- Operation in two geographically separated localities