Context & Namespaces

List All Contexts

Returns the list of contexts from the local kubectl config file

kubectl config get-contexts

Show Current Context

Returns the context currently being used by kubectl

kubectl config current-context

Change Current Context

kubectl config use-context CLUSTER_IDENTIFIER

Set “Sticky” Namespace

use a “sticky” namespace so you won’t have to specify the namespace in every command

kubectl config set-context --current -n NS_NAME

General Commands

Port Forwarding to Pod

Port-forwarding using the same port on the local machine and the target pod

kubectl port-forward -n NS_NAME POD_NAME PORT

Port-forwarding using a different port on the local machine and the target pod

kubectl port-forward -n NS_NAME POD_NAME LOCAL_PORT:TARGET_PORT

Port-forwarding to a service

kubectl port-forward -n NS_NAME svc/SERVICE_NAME PORT

Scaling

kubectl scale OBJECT OBJECT_NAME -n NS_NAME --replicas=DESIRED_COUNT

kubectl scale deployment DEPLOYMENT_NAME -n NS_NAME --replicas=5

Wait For…

Wait for Pod to become ready

kubectl wait -n NS_NAME --for=delete pod/POD_NAME --timeout=90s

Wait for Pod to become deleted

kubectl wait -n NS_NAME --for=condition=Ready pod/POD_NAME --timeout=90s

Run BASH Command On Pod

kubectl exec -n NS_NAME -it POD_NAME -- bash -c "YOUR_COMMANDS;"

SSH to Pod

kubectl exec -n NS_NAME -it POD_NAME-- bash

Force Pod Deletion

kubectl delete pod POD_NAME -n NS_NAME --grace-period=0 --force

Get Pods With Name-Only

kubectl get pods -n NS_NAME --no-headers -o custom-columns=":metadata.name"

Suspend a Cronjob

kubectl patch cronjob CRONJOB_NAME -n NS_NAME -p '{"spec" : {"suspend" : true }}'

When using Kafka, whether its a Kubernetes Operator-based (like Strimzi) or a common deployment, there are usually a set of common commands that are required for the basic day-to-day maintenance and operation of the Kafka cluster.

As a firs step you should download the Kafka distribution package. This package holds a set of Bash scripts that runs core (Java-based) Kafka components and enable interaction with the Kafka cluster (i.e creating topics, changing configs, etc.)

When running these Kafka commands, please note that they should be executed against the Zookeeper or the Bootstrap-Server depending on the Kafka version you are using.

Here is a list of commands which in my opinion are the most used and most necessary for the day-to-day operation of a Kafka cluster.

List All Topics

bin/kafka-topics.sh --list --zookeeper SERVER_ADDRESS:2181

bin/kafka-topics.sh --list --bootstrap-server SERVER_ADDRESS:9092

Show Topic Details

./bin/kafka-topics.sh --describe --bootstrap-server SERVER_ADDRESS:9092 --topic TOPIC_NAME

Create a Topic

./kafka-topics.sh --create --bootstrap-server SERVER_ADDRESS:9094 --topic TOPIC_NAME --partitions 2 --replication-factor 2

Change Topic Configuration Values

This command can be used for changing / adding configuration values to topics. This specific example shows how to change the retention ms value.

./bin/kafka-configs.sh --alter --bootstrap-server SERVER_ADDRESS:9092 --add-config retention.ms=1800000 --entity-type topics --entity-name TOPIC_NAME

Drain (purge) a Topic

./bin/kafka-configs.sh --alter --bootstrap-server SERVER_ADDRESS:9092 --add-config retention.ms=0 --entity-type topics --entity-name TOPIC_NAME

Run a Kafka Producer

The Producer process lets us add data to the topics in our Kafka cluster. When running this process, it will open a console for the producer, in which we can type inputs of data to the cluster. Each text input will be added as a separate message to the relevant topic.

bin/kafka-console-producer.sh --bootstrap-server SERVER_ADDRESS:9092 --topic TOPIC_NAME

Run a Kafka Consumer

The Consumer process lets us read data from the topics in our Kafka cluster. when using the from-begining flag we tell the Kafka cluster to let us read all the messages that are currently in the topic (even old messages that were not consumed yet).

When running this command, all read messages will be logged onto the console. If executed alongside a producer, any message that will be produced in the producer console will be logged to the console by the consumer process.

bin/kafka-console-consumer.sh --bootstrap-server SERVER_ADDRESS:9092 --topic TOPIC_NAME --from-beginning

When it comes to graph databases, no doubt that Neo4j is the shining star.

When it comes to Neo4j in production, no doubt that a Neo4j cluster is probably the best approach.

When it comes to a Neo4j cluster, in my humble opinion, there are some guidelines you should follow.

1. Make sure you never run out of disk space. This might sound trivial, but recovery from a full disk can be super-painful as it will definitely make your cluster go out of sync and may get your databases quarantined
   
2. A Causal Cluster is Causal up to a certain point. When a core server is lost, even if you had enough core servers to support the loss of that server, databases may still go offline anyway and your cluster might go out of sync.
   
3. A database might remain quarantined even after the underlying issue was resolved. Usually, databases get quarantined for a reason, and usually a justified one. However, after the underlying issue that caused the database to get quarantined is resolved, sometimes the database remains quarantined (this time with no good reason)
   
4. Have enough core servers to afford losses. The formula to calculate the number of core servers that can be lost without damaging the cluster is n=2F+1, when F is the number of servers you are willing to lose and n is the total number of required servers to achieve that.
   
5. Have a simple and efficient deployment mechanism. When it comes to clusters, especially in production, the most important and critical aspect is the day-to-day maintenance. And don’t delude yourself, you will have maintenance and lots of it. All the time. So, you must have a super-simple and standardized deployment mechanism to facilitate ad-hoc changes.  
   
6. Use a customized helm chart. Make sure you control the exact version running in production to avoid surprises. This is a general pessimistic approach I recommend taking with any 3rd party you use. You can never know when something will be changed on the vendor’s side. And you can never know how such a change will affect your application. Better safe than sorry.
   
7. Have a separate mechanism for handling volumes. Neo4j Servers are deployed via Stateful-Sets and do not allow volume size changes via the helm chart once deployed. So make sure you have a standardized deployment mechanism for facilitating that. Didn’t I already write that in section 5? Well, it’s important.
   
8. Think very carefully before you attend to a cluster issue. The first actions when attending to a cluster issue may have a huge effect on the overall outcome. Read more here.

Neo4j is (kind of) the only solution when a graph database is in need. And when it comes to production grade, in which resilience is required, a Neo4j Causal Cluster is a must.

The Neo4j Causal Cluster comprises two types of servers: 

1. Core Server – acts as read/write and forms the initial cluster. A Core Server can act as a Leader (write) or Follower (read), which varies per database.
2. Read Replica – acts as read-only and is considered disposable. A Read Replica can be removed/added with no “damage” to the cluster.

In terms of resilience, the Neo4j Causal Cluster can sustain the loss of Core-Servers. In this case, another Core-Server will become a leader for those databases led by the lost server.

Having said all that, maintaining a Neo4j Causal Cluster may not always be the easiest task. Especially when actual data is involved and data loss is not an option. So it is very important to have a basic standard approach for attending an issue in your Neo4j Causal Cluster. I would like to share with you some pointers and courses of action which I found to be very useful.

You should note that the first steps you take are super-critical and have a great effect on the overall outcome, so make sure you know exactly what you are doing and why you are doing it. Failing to do so may often result in a more severe issue than the issue which caused us to attend to the cluster in the first place.

The first step is to understand the problem

1. Go to the Neo4j UI or open cipher-shell
2. Run :sysinfo
3. Check if all databases have an online status
4. If any of the databases are offline or have exceptions or errors, try to understand what they mean (not always as trivial as it may sound)

If you were able to identify the problem, good for you! You should work to resolve it.

If you were not able to identify the problem (i.e, no errors or exceptions and all databases are online), fear not! It might be that your servers just went out of sync.

After you resolved the underlying issues (or decided its a sync issue), you should Re-Sync The Cluster

1. Put all servers in “Offline Maintenance Mode”
2. Unbind all of them from the cluster (run the unbind command on each server)
3. Take all servers out of “Offline Maintenance Mode”
4. Wait for the cluster to re-form
5. Run :sysinfo and check if all databases are online