SDP upgrade notes
Upgrade from SDP 26.7 to 26.11
A direct upgrade from SDP 26.3 to 26.11 is not supported. Upgrade to SDP 26.7 first and restart all OpenSearch Pods, see the notes for the upgrade from SDP 26.3 to 26.7 below.
Node certificates
The operator no longer accepts node certificates whose distinguished names lack domain components. Such certificates were issued up to SDP 26.3.
If you are upgrading from SDP 26.3, then you must upgrade to SDP 26.7 first and restart all OpenSearch Pods, so that they get certificates with domain components. Otherwise, the nodes which are already restarted with SDP 26.11 reject the transport connections of the nodes which are not restarted yet, and the cluster can lose its quorum during the rolling restart.
StatefulSet updates
The app.kubernetes.io/version label was removed from the listener PVC templates of the StatefulSets.
Kubernetes restricts changes to spec.volumeClaimTemplates and other StatefulSet fields, so the operator cannot apply this change in place and the following error occurs:
ERROR stackable_operator::logging::controller: Failed to reconcile object ...
unable to patch resource "opensearch-nodes-default",
ApiError: StatefulSet.apps "opensearch-nodes-default" is invalid:
spec: Forbidden: updates to statefulset spec for fields other than 'replicas', ...The OpenSearch cluster keeps running and stays healthy, so this error is easy to overlook. The operator cannot apply any further change to the StatefulSets though, for instance to the number of replicas or to the product version, until the error is resolved.
To resolve this error, you need to delete the StatefulSets right after the operator upgrade. You can do this using the following command:
kubectl delete statefulsets.apps --cascade=orphan opensearch-nodes-defaultThe operator will automatically recreate the StatefulSet.
--cascade=orphan retains the Pods, so that they are not all deleted at once.
The recreated StatefulSet adopts them and restarts them one after another to apply the new product image.
Up to SDP 26.7, the label was set to the product version, so this error occurred on every product version upgrade. Since the label is removed in SDP 26.11, the error does not occur in subsequent upgrades.
Upgrade from SDP 26.3 to 26.7
Node certificates
Since SDP 26.7, the certificates issued by the Secret Operator contain domain components in their distinguished names, and the operator adds the corresponding entry to the list of allowed node distinguished names. This list is read once at node startup, so the nodes which are still running with SDP 26.3 reject the transport connections of the nodes which are already restarted with SDP 26.7:
Node presenting certificate with SSL Principal {DC=local,DC=cluster,DC=svc,
DC=default,DC=opensearch-nodes-default-headless,DC=opensearch-nodes-default-0,
CN=generated certificate for pod} could not securely connect to the cluster.
Please ensure the principal is correct and present in the nodes_dn list.The cluster can lose its quorum during the rolling restart and be unavailable for a short time. Plan for a short downtime; the cluster recovers on its own once all Pods are restarted.
Dedicated ConfigMap for security settings
The security settings defined in the cluster specification are now stored in a separate ConfigMap named <cluster-name>-security-config.
If you used this name for your custom security configuration, then you must rename it.
Otherwise the operator will override it.
Upgrade from SDP 25.11 to 26.3
When upgrading the OpenSearch operator from SDP 25.11 to 26.3, you may encounter several warnings and errors in the operator logs.
These issues are not due to breaking changes in the Custom Resource Definition, but rather arise from incompatibilities in your configOverrides and podOverrides related to the current version.
You might have configured the OpenSearch security plugin, TLS and other features with overrides which are now handled directly by the operator.
If configOverrides replace values that are already set by the operator, a corresponding warning will be issued:
WARN reconciling object{...}: stackable_opensearch_operator::controller::build::node_config:
configOverrides: Configuration setting "plugins.security.ssl.http.pemcert_filepath" changed
from "/stackable/opensearch/config/tls/server/tls.crt" to
"/stackable/opensearch/config/tls/tls.crt".TLS
The operator now offers the option to configure TLS using SecretClasses:
spec:
clusterConfig:
tls:
internalSecretClass: tls
serverSecretClass: tlsThis means that the following configOverrides are now obsolete and must be removed:
configOverrides:
opensearch.yml:
plugins.security.ssl.http.enabled: ...
plugins.security.ssl.http.pemcert_filepath: ...
plugins.security.ssl.http.pemkey_filepath: ...
plugins.security.ssl.http.pemtrustedcas_filepath: ...
plugins.security.ssl.transport.enabled: ...
plugins.security.ssl.transport.pemcert_filepath: ...
plugins.security.ssl.transport.pemkey_filepath: ...
plugins.security.ssl.transport.pemtrustedcas_filepath: ...Additionally, the following podOverrides for the TLS volumes and volumeMounts must be removed:
podOverrides:
spec:
containers:
- name: opensearch
volumeMounts:
- mountPath: /stackable/opensearch/config/tls
name: tls
volumes:
- name: tls
ephemeral:
volumeClaimTemplate:
metadata:
annotations:
secrets.stackable.tech/class: tls
secrets.stackable.tech/scope: ...
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "1"
storageClassName: secrets.stackable.techRefer to the TLS documentation for further details.
Security plugin configuration
The configuration of the OpenSearch security plugin is now managed by the operator.
You may have defined the configuration files in a Secret or ConfigMap.
These files were previously used to initialize the security index when you set plugins.security.allow_default_init_securityindex to true.
Since these files are not utilized after the security index creation, you can safely remove the corresponding configOverrides and podOverrides, particularly to prevent mount path conflicts:
configOverrides:
opensearch.yml:
plugins.security.allow_default_init_securityindex: "true"
podOverrides:
spec:
containers:
- name: opensearch
volumeMounts:
- mountPath: /stackable/opensearch/config/opensearch-security
name: security-config
readOnly: true
volumes:
- name: security-config
secret:
defaultMode: 432
secretName: opensearch-security-configPlease have a look at the documentation on security settings, if you want the operator to manage certain security configurations.
StatefulSet updates
The OpenSearch operator deploys a StatefulSet, in which Kubernetes restricts changes to certain fields. As a result, the following error may occur:
ERROR stackable_operator::logging::controller: Failed to reconcile object ...
unable to patch resource "opensearch-nodes-default",
ApiError: StatefulSet.apps "opensearch-nodes-default" is invalid:
spec: Forbidden: updates to statefulset spec for fields other than 'replicas', ...To resolve this error, you need to delete the StatefulSets. You can do this using the following command:
kubectl delete statefulsets.apps --cascade=orphan opensearch-nodes-defaultThe operator will automatically recreate the StatefulSet.
Service discovery
The OpenSearch operator now deploys a discovery ConfigMap that is named after the OpenSearch cluster:
apiVersion: v1
kind: ConfigMap
metadata:
name: opensearch
data:
OPENSEARCH_HOSTNAME: opensearch.default.svc.cluster.local
OPENSEARCH_HOSTS: https://opensearch.default.svc.cluster.local:9200
OPENSEARCH_PORT: "9200"
OPENSEARCH_PROTOCOL: httpsConsider using this ConfigMap in your OpenSearch clients instead of hard-coding a Service or IP address.
For example, the Helm values for OpenSearch Dashboards can reference the discovery ConfigMap:
opensearchHosts: null # Use the discovery ConfigMap instead
extraEnvs:
- name: OPENSEARCH_HOSTS
valueFrom:
configMapKeyRef:
name: opensearch
key: OPENSEARCH_HOSTSPlease consult the Discovery documentation for instructions on how to select the role groups exposed by the discovery ConfigMap.