Spring Cloud Data Flow (SCDF) is a cloud-native orchestration service for composing microservice based applications on modern runtimes. With Spring Cloud Data Flow, developers can create and orchestrate data pipelines for common use cases such as data ingestion, real-time analytics, and data import/export.
- Develop applications using DSL, REST-APIs, Dashboard, and the drag-and-drop GUI - Flo
- Create, unit-test, troubleshoot and manage microservice applications in isolation
- Build data pipelines rapidly using out-of-the-box stream and task/batch components
- Consume microservices written in Java or other languages as docker artifacts
- Scale data pipelines without interrupting data flows
- Orchestrate data-centric applications on a variety of modern runtime platforms including Cloud Foundry, Apache YARN, Apache Mesos, and Kubernetes
- Take advantage of metrics, health checks, and the remote management of each microservice application
Deploying .NET Components to SCDF
Steeltoe Stream enables .NET components to run on SCDF by parsing and acting on the appropriate SCDF configuration parameters and binding the .NET application to the underlying message broker. .NET components built with Steeltoe are interoperable with their Java counterparts giving you the ability to leverage prebuilt components written in other languages (such as Java) and maintained by the Spring Cloud Data Flow team. To learn more about these components, SCDF team has an excellent guide that describes installing and deploying Java based applications on SCDF at the Spring Cloud Data Flow Microsite. We encourage you to get familiar with that content.
This guide is focused on describing how you can deploy .NET based Stream components to SCDF and how you can easily combine them with already existing components written in other languages such as Java.
Steeltoe Stream Processing with Data Flow and RabbitMQ
This section shows how to register stream applications with Data Flow, create a Stream DSL, and deploy the resulting application to Cloud Foundry, Kubernetes, and your local machine.
In our previous guides we created
ISink .NET streaming components using Steeltoe and deployed them as standalone applications (i.e. not on SCDF).
In this guide, we describe how you can register these Steeltoe based components with Data Flow, combine them with other Java based components, create a Stream DSL to orchestrate their interactions, and deploy the final application to Cloud Foundry.
The samples repo has a number of Steeltoe Stream sample applications. For this example we will be using the
CloudDataFlowLogToUpperProcessor sample, which implements an
IProcessor interface and the
CloudDataFlowLogSink sample, which implements an
ISource use the
HttpSource application available at
Spring Cloud team publishes a number of sample applications as
docker artifacts at the
https://repo.spring.io Maven repository. These pre-packaged applications can be used with Steeltoe Stream.
The Data Flow Dashboard
Assuming Data Flow is installed and running , open your browser at
<data-flow-url> depends on the platform. See the installation guide to determining the base URL for your installation.
Applications in Spring Cloud Data Flow are registered as named resources so that they may be referenced when you use the Data Flow DSL to configure and compose streaming pipelines. Registration associates a logical application name and type with a physical resource, which is given by a URI.
The URI conforms to a schema.
As of Steeltoe 3.1.0-rc1 only Docker images are supported.
Data Flow defines some logical application types to indicate its role as a streaming component, a task, or a standalone application. For streaming applications, as you might expect, we use
The Data Flow Dashboard lands on the Application Registration view, where we can register the source, processor, and sink applications, as follows:
In this step, we register the applications we previously created. When you register an application, you provide its:
- Location URI (Maven, HTTP, Docker, file, and so on)
- Application version
- Application type (source, processor, or sink)
- Application name
The following table shows the applications listed in the Samples:
|App Name||App Type||App URI|
This example assumes Spring Cloud Data Flow and Skipper servers are running in your Cloud Foundry environment.
In addition to the samples, register the pre-packaged
HttpSource source application. To do so:
From the Applications view, select ADD APPLICATION(S). This shows a view that lets you register applications.
Select Register one or more applications and enter the
URIfor the source application.
dockerartifact of the
http-source, as follows:
Click on NEW APPLICATION to display another instance of the form to enter the values for the processor.
dockerartifact of the
BasicStreamProcessorprocessor application named
basicstreamprocessor, as follows:
Click on NEW APPLICATION to display another instance of the form to enter the values for the sink.
dockerartifact of the
BasicStreamSinksink application named
basicstreamsink, as follows:
Click on IMPORT APPLICATION(S) to complete the registration. Doing so takes you back to the Applications view, which lists your applications. The following image shows an example:
Creating the Stream Definition
To create the stream definition:
- Select Streams from the left navigation bar. This shows the main Streams view, as follows:
- Select Create stream(s) to display a graphical editor to create the stream definition, as the following image shows:
You can see the
Sink applications, as registered above, in the left panel.
Drag and drop each application to the canvas and then use the handles to connect them together. Notice the equivalent Data Flow DSL definition in the top text panel.
You can type the name of the stream
basicsteeltoestreamwhen creating the stream.
To deploy your stream,
Click on the arrow head icon to deploy the stream. Doing so takes you to the Deploy Stream page, where you may enter additional deployment properties.
Deploy, as follows:
- When deploying the stream, choose the target platform accounts from local, Kubernetes, or Cloud Foundry. This is based on the Spring Cloud Skipper server deployer platform account setup.
When all the applications are running, the stream is successfully deployed.
Before registering and deploying stream applications to Cloud Foundry by using the instructions shown earlier, you should ensure that you have an instance of Spring Cloud Data Flow running on Cloud Foundry. Follow the Cloud Foundry installation guide for reference.
Once you have followed the steps shown earlier in this chapter and have registered the applications as well as deployed the stream, you can see the successfully deployed applications in your in your Org and Space in Cloud Foundry.
You can access the runtime information of your stream applications in the Spring Cloud Data Flow dashboard as well.
Besides verifying the runtime status of your stream, you should also verify the logging output produced by the
basicstreamsink sink. In Cloud Foundry Apps Manager, click the Logs tab of the
basicstreamsink sink application.
The logging statements should look like the following:
Comparison with Standalone Deployment
In this section, we deployed the stream by using Spring Cloud Data Flow with the stream DSL:
http | steeltoebasicprocessor | steeltoebasicsink
When these three applications are deployed as standalone applications, you need to set the binding properties that connect the applications to make them into a stream.
Instead, Spring Cloud Data Flow lets you deploy all three streaming applications as a single stream by taking care of the plumbing of one application to the other to form the data flow.